Satellite beam coverage adjustment method and system based on reinforcement learning

Abstract

The application provides a satellite beam coverage adjustment method based on reinforcement learning, which comprises initial allocation, optimization adjustment and scheme generation; in the initial allocation, relevant parameters are input, users are sorted and allocated according to the comprehensive weighting priority, and the unallocated users form a to-be-optimized set; if the to-be-optimized set is not empty, the optimization adjustment is entered, an agent is configured for the beam, and a reinforcement learning environment is initialized, and the optimization is circularly executed: the agent adjusts the beam parameters, determines the covered users, processes the multi-beam and super-capacity problems, calculates the final reward, stores the experience to update the network, and adds a new beam if necessary; and finally, the final beam coverage scheme is output. Therefore, the beam center is determined according to the task priority, the influence of the initial input on the result is avoided, and the resource utilization is improved through the dynamic adjustment of the beam by the reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of satellite communication technology, and in particular to a satellite beam coverage adjustment method and system based on reinforcement learning. Background Technology

[0002] The next generation of high-throughput satellites will achieve throughputs of hundreds of gigabits per second or even terabits per second, with the number of beams increasing from dozens to hundreds. In practical applications, communication between satellites and various user terminals, such as those mounted on vehicles and ships, as well as fixed ground stations, presents challenges to the overall scheduling and optimization of different satellite resources. Traditional high-throughput satellites have issues such as regional limitations in meeting user needs and resource waste.

[0003] Traditional beamforming methods have three main problems.

[0004] 1. Since traditional satellites cannot adjust beam resources while in orbit, in order to ensure that users can access the service from different geographical locations, the beam needs to cover the entire service area as completely as possible. However, the distribution of users is not uniform, and not all beams can have users online continuously. Some beams are located in remote areas and have no service requests for a long time, resulting in wasted resources.

[0005] Second, due to uneven user distribution, some beams in densely populated user areas may be rejected due to lack of resources.

[0006] Third, due to various constraints, the beam coverage area may not completely encompass the entire service area, and some users may not be within the coverage area and therefore unable to communicate.

[0007] With the application of flexible payloads such as active beamforming networks and phased array antennas, the adjustability of beam coverage means that coverage no longer needs to pursue geographical coverage satisfaction. Beams only need to provide flexible coverage in areas with user access. When adjusting the coverage area, it is necessary to dynamically optimize parameters such as the number, size, and center position of beams based on the spatiotemporal distribution of services and channel conditions. The expansion of resource dimensions and the increase in the scale of adjustable parameters significantly increase the resource configuration complexity of such systems. Therefore, for next-generation high-throughput satellite systems, it is necessary to study the design of optimization algorithms to improve system resource utilization and user service quality under the constraints of on-board flexible antennas.

[0008] Existing methods for planning and adjusting beam coverage areas for high-throughput satellites have three significant drawbacks:

[0009] (1) Current coverage planning algorithms consider continuous beam coverage. In order to meet geographical coverage satisfaction, they cover many areas where no users exist, resulting in a waste of some beam resources.

[0010] (2) In current research on coverage planning algorithms, most of them only take the user's priority as the sorting basis for beam allocation, without taking into account the priority of the user's business at different times, which makes some low-frequency but high-importance businesses not well valued.

[0011] (3) In current research on coverage planning algorithms, random methods are often used to select the beam center before subsequent optimization. The selection of the initial center point has a significant impact on the final result. Summary of the Invention

[0012] To overcome the aforementioned technical deficiencies, the present invention aims to provide a satellite beam coverage adjustment method and system based on reinforcement learning. Based on reinforcement learning, and considering the dynamic changes of multiple users, this invention conducts research on beam coverage planning methods to improve the efficiency of satellite resource management.

[0013] To achieve the aforementioned technical effects, this invention provides a satellite beam coverage adjustment method based on reinforcement learning, comprising initial allocation, optimization adjustment, and scheme generation steps:

[0014] The initial allocation includes:

[0015] Input satellite parameters, user parameters, beam parameters, and optimized control parameters;

[0016] Users are sorted based on their comprehensive weighting priority to form a set to be assigned.

[0017] Traverse the beams and assign users to them. Users that meet the coverage conditions are assigned to the corresponding beams, and unassigned users form a set to be optimized. If the set to be assigned is empty, proceed to the scheme generation step; otherwise, proceed to the optimization and adjustment step.

[0018] The optimization adjustments include:

[0019] Each beam is configured with a corresponding agent, and the reinforcement learning environment is initialized based on the initial allocation results. The initial state of the agent includes the corresponding beam state and the set to be optimized.

[0020] Optimization of loop execution a preset number of times:

[0021] (a) The agent performs adjustment actions based on the current state through a reinforcement learning network to optimize beam coverage parameters;

[0022] (b) Re-determine the users covered by each beam, select the beam to which the multi-beam coverage users belong based on the principle of quality of service priority, and make user adjustments for the overcapacity beam to meet the capacity constraints;

[0023] (c) Calculate the final reward that integrates local and global rewards, store the experience including state, action, the final reward and the next state, and update the agent network parameters based on the experience;

[0024] (d) Update beam status. If there are unassigned users and the maximum number of beams has not been reached, add a new beam.

[0025] The scheme generation includes:

[0026] The final beam coverage scheme is output based on the allocation and optimization results.

[0027] Optionally, the satellite parameters include the satellite nadir position and satellite altitude; the user parameters include a user heatmap matrix and a user demand capacity satisfaction rate matrix, wherein the user heatmap matrix includes the user comprehensive weighted priority; the beam parameters include a beam matrix, beamwidth angle vector matrices, beampower vectors, available frequency bands, and beam gain vectors; and the optimization control parameter is the total number of times the optimization loop is executed.

[0028] Optionally, in the initial allocation, after sorting users based on the comprehensive weighted priority of users, when traversing beams to perform user allocation, for the current beam b, if the highest priority user in the set to be allocated has not been allocated and the angle between the line connecting the user to the satellite and the line connecting the center point of beam b to the satellite is greater than the half beam width of beam b, then the user is allocated to beam b and set as the center point of beam b, and other users to be allocated within the half beam width coverage area of ​​the beam are assigned to beam b.

[0029] Optionally, the beam state in the initial state of the agent includes the center point position, width, user set to which it belongs, and available capacity of the corresponding beam after initial allocation; the set to be optimized includes the location and required capacity of unallocated users.

[0030] Optionally, in the optimization adjustment, the adjustment actions performed by the agent include adjusting the beam center direction and beam width, and the reinforcement learning network is a DQN network.

[0031] Optionally, in the optimization adjustment, the service quality priority principle for selecting the appropriate beam for multi-beam coverage users includes:

[0032] Select the beam that meets the capacity requirements and has the strongest signal;

[0033] The user adjustment method for the supercapacity beam is as follows: users are removed from the beam in order of user priority from low to high until the total user demand capacity within the beam does not exceed the available capacity.

[0034] Optionally, in the optimization adjustment, the experience includes the agent's state before adjustment, the actions performed, the final reward integrating local and global rewards, and the next state after adjustment;

[0035] The method for updating network parameters based on the experience is as follows: sample batches of data from the experience to update the network parameters of the agent.

[0036] Optionally, the rules for creating new beams in the optimization adjustment are as follows:

[0037] If the set to be optimized still contains unassigned users and the existing number of beams is less than the total number of beams, then a new beam is created with the position of the highest priority user among the unassigned users as the center point.

[0038] Optionally, the local reward for the b-th agent is:

[0039] r agent,b =w1·η coverage,b +w2·η capacity,b -w3·L interrupt,b +w4·r newuser,b ;

[0040] Where, η coverage,b To determine the coverage satisfaction rate, η capacity,b For capacity fulfillment rate, L interrupt,b For interruption losses, r newuser,b As an additional reward for new users after beam b covers the area, w i (i = 1, 2, 3, 4) represents the weights of each item;

[0041] The global reward is:

[0042] r global =W1·R coverage +W2·R capacity +W3·n saved ;

[0043] Among them, R coverage For global coverage, R capacity For global capacity utilization, n saved To save beamwidth; W i (i = 1, 2, 3) represents the weights of each item;

[0044] The final reward is:

[0045]

[0046] Among them, W agent and W global These are the weights corresponding to the local reward and the global reward, respectively, n b This represents the total number of beams.

[0047] On the other hand, based on the same inventive structure, the present invention also provides a satellite beam coverage adjustment system based on reinforcement learning, the system being used to implement the satellite beam coverage adjustment method based on reinforcement learning as described above.

[0048] This invention divides the beam coverage adjustment method into two main parts: initial allocation and optimization adjustment. First, initial beam allocation is performed for all users based on initial conditions. Then, multi-agent DQN (each beam is considered an agent) is used for decision-making. Thus, this invention determines the beam center according to task priority, avoiding the influence of initial input on the result; and improves resource utilization by dynamically adjusting the beam through reinforcement learning. Attached Figure Description

[0049] Figure 1 A flowchart illustrating the steps of a satellite beam coverage adjustment method based on reinforcement learning, as provided in an embodiment of the present invention.

[0050] Figure 2 The flowchart below shows the initial allocation of a satellite beam coverage adjustment method based on reinforcement learning, as provided in an embodiment of the present invention.

[0051] Figure 3 The following is a flowchart illustrating the optimization adjustment of a satellite beam coverage adjustment method based on reinforcement learning, as provided in an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram of the user coverage situation in the lower hemisphere projection of latitude and longitude coordinates in a specific application example of the satellite beam coverage adjustment method based on reinforcement learning provided in an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of user coverage under UV coordinates in a specific application example of the satellite beam coverage adjustment method based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0056] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0057] Before describing the embodiments of this application in detail, the technical concept of this application is first briefly described: This invention is aimed at the communication environment between GEO (Geostationary Earth Orbit) satellites, mobile user terminals, and fixed ground stations. It adopts the idea of ​​"initial allocation + reinforcement learning optimization and adjustment". The initial allocation determines the initial beam coverage according to the comprehensive priority of users (including service urgency, etc.). In the optimization and adjustment stage, each beam is treated as an intelligent agent to dynamically adjust the beam parameters. Combined with local and global reward update models, the resource utilization and service quality are improved.

[0058] The specific principles of the reinforcement learning-based satellite beam coverage adjustment method of this application will be described below with reference to specific embodiments.

[0059] Figure 1 This invention illustrates a satellite beam coverage adjustment method based on reinforcement learning, comprising the steps of S101 initial allocation, S102 optimization adjustment, and S103 scheme generation.

[0060] S101: Input satellite parameters, user parameters, beam parameters and optimization control parameters; sort users based on user comprehensive weighting priority to form a set to be assigned; traverse beams and assign users to them, assign users that meet the coverage conditions to the corresponding beams, and form an optimization set for unassigned users; if the set to be assigned is empty, proceed to step S103, otherwise proceed to step S102.

[0061] The satellite parameters include the satellite's nadir position and altitude; the user parameters include the user heatmap matrix and the user demand capacity satisfaction rate matrix, with the user heatmap matrix containing the user's comprehensive weighted priority; the beam parameters include the beam matrix, beamwidth angle vector matrix, beampower vector, available frequency band, and beam gain vector; the optimization control parameter is the total number of times the optimization loop is executed.

[0062] Furthermore, in the initial allocation, after sorting users based on the comprehensive weighting priority, when traversing beams to perform user allocation, for the current beam b, if the highest priority user in the set to be allocated has not been allocated and the angle between the line connecting the user to the satellite and the line connecting the center point of beam b to the satellite is greater than half the beam width of beam b, then the user is allocated to beam b and set as the center point of beam b, and other users to be allocated within the half-width coverage area of ​​the beam are assigned to beam b.

[0063] For the current beam b, determine whether the highest priority user in the set to be assigned has not been assigned. Simultaneously, calculate the angle between the line connecting the user to the satellite and the line connecting the center point of beam b (which can be initially empty, but the first user's location is used as a reference during the initial assignment). If the angle is greater than half the beamwidth of beam b, it means the user is not within the current coverage area of ​​the beam. In this case, assign the user to beam b, using its location as the center point of beam b. This avoids coverage deviations caused by random initial beam center selection, ensuring accurate coverage for high-priority users. At the same time, other users to be assigned within the half-width coverage area of ​​this beam (i.e., users whose angle between the line connecting the user to the satellite and the line connecting the center point of beam b to the satellite is ≤ half the beamwidth) are assigned to beam b. This prioritizes high-priority users while quickly incorporating surrounding users, reducing invalid coverage in user-less areas and laying an efficient initial coverage foundation for subsequent optimization and adjustments.

[0064] S102: Configure corresponding agents for each beam, initialize the reinforcement learning environment based on the initial allocation results, and the initial state of the agent includes the corresponding beam state and the set to be optimized.

[0065] Specifically, the state of each agent Including the current beam status (center pointing to location) b =(long b ,lat b ), the current user set CoveredUser of the beam b Beam remaining capacity C b The status of unassigned users within a certain range (location) u =(long u,lat u ), demand capacity c u Actions of the intelligent agent. This includes adjusting the beam center pointing (azimuth angle Δu and elevation angle Δv adjusted from a satellite perspective), with adjustment ranges limited to min u ≤ u + Δu ≤ maxu, minv ≤ v + Δv ≤ max v; and adjusting the beamwidth ΔBW, with adjustment ranges limited to min BW ≤ BW + ΔBW ≤ max BW. The reward function for each agent includes the ratio of covered users to their assigned users, beam capacity utilization, and interruption losses. The global reward function r includes overall coverage satisfaction rate, overall capacity satisfaction rate, and beam savings.

[0066] Assigning corresponding agents to each beam is a key design element of the distributed reinforcement learning framework. Each beam corresponds to an independent agent, enabling it to autonomously perceive its own coverage status and make adjustments, while simultaneously achieving collaborative optimization through a global reward mechanism. The initialization of the reinforcement learning environment is based on the initial allocation results. The initial state of the agent includes: the current center point position of the beam after the initial allocation (latitude and longitude or azimuth / elevation angle), beamwidth, the set of users to which it belongs and the capacity requirements of each user, the remaining available capacity of the beam, and the set to be optimized (users not covered after the initial allocation, their locations, capacity requirements, etc.). This state definition ensures that the agent can fully perceive its own coverage performance and the optimization objectives.

[0067] Optimization by looping the code a preset number of times (N times):

[0068] (a) The agent performs adjustment actions based on the current state through a reinforcement learning network to optimize beam coverage parameters. These adjustments include adjusting the beam center direction and beamwidth. The reinforcement learning network is a DQN network. Specifically, the agent's adjustment actions based on the current state through the reinforcement learning network (such as a DQN network) involve the reinforcement learning network learning historical experience and outputting an adjustment strategy for the beam coverage parameters. Specific actions include adjusting the azimuth offset, elevation offset, and beamwidth of the beam center. This allows the beam to dynamically adapt to changes in user distribution, avoiding the static limitations of initial allocation.

[0069] (b) Re-determine the users covered by each beam, select the appropriate beam for multi-beam coverage users based on the quality of service (QoS) priority principle, and adjust the users of overcapacity beams to meet capacity constraints. Specifically, the QoS priority principle for selecting the appropriate beam for multi-beam coverage users in the optimization adjustment includes: selecting the beam with the strongest signal that meets the capacity requirements; an overcapacity beam refers to a beam whose total user demand capacity within the beam is greater than the available capacity of the beam; the user adjustment method for overcapacity beams is: removing users from the beam according to user priority from low to high until the total user demand capacity within the beam does not exceed the available capacity. After adjustment, the coverage relationship between users and beams needs to be recalculated (specifically determined by whether the angle between the user-to-satellite connection and the beam center connection is less than or equal to half the beamwidth). For users covered by multiple beams simultaneously, the appropriate beam is selected based on the principle of quality of service priority: the beam with signal strength that meets the communication threshold and whose remaining capacity can meet the user's needs is selected first. If there are multiple options, the one with the strongest signal is selected to ensure the best communication quality for the user. For overcapacity beams where the total demand capacity of the user exceeds the available capacity of the beam, the users are removed from the beams in order of increasing overall user weighting priority until the total demand capacity is less than or equal to the available capacity, so as to avoid service degradation caused by beam overload.

[0070] (c) Calculate the final reward integrating local and global rewards, store the experience including state, action, the final reward, and the next state, and update the agent's network parameters based on the experience; the experience includes the agent's state before adjustment, the action performed, the final reward integrating local and global rewards, and the next state after adjustment; the method for updating network parameters based on the experience is to sample batches of data from the experience to update the agent's network parameters. To balance single-beam performance and overall system performance, this embodiment adopts a dual feedback mechanism of "local reward + global reward". Local reward reflects the coverage effect of a single beam (such as the number of covered users, capacity utilization) and interruption loss (such as user dropout penalty due to adjustment); global reward reflects the overall system coverage (percentage of unallocated users), resource utilization (total capacity utilization efficiency), and beam saving (avoiding unnecessary beam addition). The two are integrated into the final reward through weights (which can be dynamically adjusted according to the actual scenario), serving as a quantitative indicator for evaluating the quality of the current action. Subsequently, the complete experience of "current state - action performed - final reward - next state" is stored in the experience replay pool. The network parameters of the agent are updated by randomly sampling batches of experience. The prediction error is minimized by using the temporal difference algorithm, so that the network gradually learns "which action to perform in a specific state to obtain a higher reward", thereby achieving policy optimization.

[0071] In this embodiment, the local reward of the b-th agent is defined as:

[0072] r agent,b =w1·η coverage,b+w2·η capacity,b -w3·L interrupt,b +w4·r newuser,b ;

[0073] Where, η coverage,b To determine the coverage satisfaction rate, η capacity,b For capacity fulfillment rate, L interrupt,b For interruption losses, r newiser,b As an additional reward for new users after beam b covers the area, w i (i = 1, 2, 3, 4) represents the weights of each item;

[0074] The global reward is:

[0075] r global =W1·R coverage +W2·R capacity +W3·n saved ;

[0076] Among them, R coverage For global coverage, R capacity For global capacity utilization, n saved To save beamwidth; W i (i = 1, 2, 3) represents the weights of each item;

[0077] The final reward is:

[0078]

[0079] Among them, W agent and W global These are the weights corresponding to the local reward and the global reward, respectively, n b This represents the total number of beams.

[0080] (d) Update beam status. If there are unassigned users and the maximum number of beams has not been reached, add a new beam. The rule for creating a new beam is as follows: if there are still unassigned users in the set to be optimized and the current number of beams has not reached the total number of beams, then create a new beam with the position of the highest priority user among the unassigned users as the center point. That is, after adjustment, update the beam status (including the new center point, width, user set, and remaining capacity). If there are still unassigned users in the set to be optimized and the current total number of beams has not reached the system maximum limit, then create a new beam with the position of the user with the highest overall weighted priority among the unassigned users as the center point, ensuring that high-priority users are not missed, while avoiding excessive resource consumption through the maximum number of beams constraint.

[0081] Through the above N iterations of optimization, the beam coverage scheme can dynamically adapt to user distribution while meeting user priority and capacity constraints, thereby achieving a dual improvement in resource utilization and service quality.

[0082] The optimization and adjustment phase described in this embodiment is the core process of dynamically optimizing beam coverage through reinforcement learning based on the initial allocation. It aims to solve problems such as coverage deviation, resource waste, or insufficient user coverage that may exist in the initial allocation.

[0083] S103: Output the final beam coverage scheme based on the allocation and optimization results.

[0084] See Figure 2 The specific process for the initial allocation is as follows:

[0085] (1) Initialize the parameters, where:

[0086] The user heatmap matrix USER includes n u Information for each user, with each column containing the user's ID (u) and the longitude of the user's location (long). u latitude u The communication capacity required by the user, c u and user comprehensive weighting priority rank u u∈{1,2,…,n u}

[0087] The beam matrix BEAM includes n b Information for each beam, with each column representing the longitude of the beam center point. b latitude b The available capacity of the beam C b b∈{1,2,…,n b All parameters are initialized to null.

[0088] User demand capacity fulfillment rate matrix M satisfied Initialize to 0.

[0089] Beamwidth angular vector matrix BW, BW b If the number is positive, all elements are initialized to BW. min b∈{1,2,…,n b}

[0090] The power vector P of each beam, whose elements are the current power P of each beam. b (0≤P b ≤P max b∈{1,2,…,n} b}

[0091] The available frequency band is [d1,d2], where d1 and d2 are positive, the unit is MHz, and d2 is greater than d1.

[0092] Each beam gain vector G, with elements G b b∈{1,2,…,nb}

[0093] Adjust and optimize the total number of loop executions N.

[0094] (2) Sort users based on the comprehensive weighting priority to form a set to be assigned; specifically, arrange all users in order of priority from high to low to generate a set to be assigned; the comprehensive weighting priority in this embodiment is set based on user level, business urgency and historical value contribution, and the comprehensive weighting priority of different users is different.

[0095] (3) Select the user u with the highest priority from the set to be assigned as the filtering object.

[0096] (4) Traverse the beam (beam number b = 1 to n) b ), determine whether the beam number b is 1, if so, proceed to step (7); otherwise, proceed to step (5).

[0097] (5) Select the center point of the beam currently being traversed in sequence, and calculate the angle θ between the line connecting the center point to the satellite and the line connecting the selected object u to the satellite.

[0098] (6) Determine whether θ is greater than the half beamwidth of the corresponding beam. If so, proceed to step (7). Otherwise, remove the current filter object from the set to be assigned and then proceed to step (11).

[0099] (7) Substitute the filtered object u to beam b and take its position as the center point of beam b, and remove user u from the set to be assigned.

[0100] (8) Select other users in the set to be assigned in turn, and calculate the angle θ between the line connecting them to the satellite and the line connecting the beam center to the satellite.

[0101] (9) Determine whether θ is greater than the half beamwidth of beam b. If not, proceed to step (10). If yes, return to step (8).

[0102] (10) Determine whether user u is the highest priority user in the set to be assigned. If yes, proceed to step (11); otherwise, return to step (8).

[0103] (11) Determine whether the beam number b is equal to the total number of beams n. b If yes, proceed to step (13); otherwise, proceed to step (12).

[0104] (12) Beam number b = b + 1, and then repeat step (4).

[0105] (13) After the initial allocation is completed, determine whether there are any unallocated users. If so, select the users in the user set who do not belong to the beam as the set to be optimized and proceed to the next step; otherwise, proceed to step (27).

[0106] Further, see Figure 3 The specific process for optimization and adjustment is as follows:

[0107] (14) Initialize the reinforcement learning environment. Each beam is a single agent with the state of the beam after initial allocation (center, width, user, remaining capacity) and the set of unallocated users.

[0108] (15) Adjust and optimize the number of runs t = 1.

[0109] (16) Each beam agent selects and executes an action (adjusting beam center direction and beam width) based on the current state.

[0110] (17) Calculate the users covered by each beam: including previously assigned users and unassigned users.

[0111] (18) For users covered by multiple beams, select the beam that meets the capacity requirements and has the strongest signal as their assigned beam.

[0112] (19) Check the capacity constraints of each beam. If the total user demand capacity within the beam is greater than the available capacity of the beam, remove users from the beam in order of priority from low to high until the capacity constraints are met.

[0113] (20) Calculate rewards (including local rewards, global rewards and final rewards).

[0114] (21) Storage experience

[0115] (22) Update beam status (beam center, width, user, remaining capacity) and unassigned users.

[0116] (23) Update the DQN network by sampling batches.

[0117] (24) Determine if there are still unassigned users. If so, and the current number of beams has not reached the total number of beams n. b A new beam can be created (based on the initial allocation, a beam is created centered on the highest priority user among the unallocated users, with an initial beamwidth of BW). min Otherwise, proceed to the next step.

[0118] (25) Determine whether the number of times of adjustment and optimization t has reached the maximum total number of times N. If yes, proceed to step (27); otherwise, proceed to the next step.

[0119] (26) t = t + 1, then repeat step (16).

[0120] (27) Output the solution and end the loop.

[0121] In a specific application example, the scenario is set as a GEO satellite with a nadir of 117°E, 0°N, the number of loops N is 300, and the number of users n b The value is set to 2000, with a loop length of 300 times and 200 time steps per loop. The beam coverage result for each round is calculated using the method described in this embodiment, as follows: Figures 4-5 As shown; compared with the prior art, the present invention uses a coverage area adjustment method based on reinforcement learning to dynamically adjust the beam coverage area according to changes in user needs, using narrow beam coverage for hotspot areas and wide beam coverage for non-hotspot areas, thereby improving the efficiency of beam resource utilization; furthermore, by introducing task priority to determine the center position of the beam, the influence of the initial input on the final solution is effectively avoided.

[0122] In another embodiment, a reinforcement learning-based satellite beam coverage adjustment system is also provided. This system implements the reinforcement learning-based satellite beam coverage adjustment method as described in the above embodiments. Specifically, the system can execute corresponding operation procedures through various functional modules to implement the method shown in the above embodiments. The corresponding operation procedures are as described above and will not be repeated here.

[0123] In summary, this invention addresses the problem of traditional user-leveling methods in communication satellite mission planning neglecting high-importance services from low-priority users by introducing a multi-dimensional weighted dynamic evaluation model. The model establishes a comprehensive weighted evaluation and ranking method based on "Comprehensive Weighted Priority = User Level + Service Urgency + Historical Value Contribution," avoiding the issue of time-sensitive services receiving insufficient attention due to low user levels. Furthermore, it improves resource utilization by dynamically adjusting beams through reinforcement learning.

[0124] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0125] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A satellite beam coverage adjustment method based on reinforcement learning, characterized in that, This includes initial allocation, optimization and adjustment, and solution generation steps: The initial allocation includes: Input satellite parameters, user parameters, beam parameters, and optimized control parameters; Users are sorted based on their comprehensive weighting priority to form a set to be assigned. Traverse the beams and assign users to them. Users that meet the coverage conditions are assigned to the corresponding beams, and unassigned users form a set to be optimized. If the set to be assigned is empty, proceed to the scheme generation step; otherwise, proceed to the optimization and adjustment step. The optimization adjustments include: Each beam is configured with a corresponding agent, and the reinforcement learning environment is initialized based on the initial allocation results. The initial state of the agent includes the corresponding beam state and the set to be optimized. Optimization of loop execution a preset number of times: (a) The agent performs adjustment actions based on the current state through a reinforcement learning network to optimize beam coverage parameters; (b) Re-determine the users covered by each beam, select the beam to which the multi-beam coverage users belong based on the principle of quality of service priority, and make user adjustments for the overcapacity beam to meet the capacity constraints; (c) Calculate the final reward that integrates local and global rewards, store the experience including state, action, the final reward and the next state, and update the agent network parameters based on the experience; In the initial allocation, after sorting users based on the comprehensive weighting priority, when traversing beams to perform user allocation, for the current beam b, if the highest priority user in the set to be allocated has not been allocated and the angle between the line connecting the user to the satellite and the line connecting the center point of beam b to the satellite is greater than the half beam width of beam b, then the user is allocated to beam b and set as the center point of beam b, and other users to be allocated within the coverage area of ​​the half beam width are assigned to beam b. The local reward for the b-th agent is: ； in, To achieve coverage satisfaction rate, For capacity fulfillment rate, For interruption losses, Additional rewards for new users after beam b coverage. The corresponding weights for each item; The global reward is: ； in, For global coverage, For global capacity utilization, To save beamwidth; The corresponding weights for each item; The final reward is: ； in, and These are the weights corresponding to the local reward and the global reward, respectively. Total number of beams; (d) Update beam status. If there are unassigned users and the maximum number of beams has not been reached, add a new beam. The scheme generation includes: The final beam coverage scheme is output based on the allocation and optimization results.

2. The satellite beam coverage adjustment method based on reinforcement learning according to claim 1, characterized in that, The satellite parameters include the satellite's nadir position and altitude; the user parameters include a user heatmap matrix and a user demand capacity satisfaction rate matrix, wherein the user heatmap matrix contains the user's comprehensive weighted priority; the beam parameters include a beam matrix, beamwidth angle vector matrices, beampower vectors, available frequency bands, and beam gain vectors; the optimization control parameter is the total number of times the optimization loop is executed.

3. The satellite beam coverage adjustment method based on reinforcement learning according to claim 1, characterized in that, The beam state in the initial state of the agent includes the center point position, width, user set to which it belongs, and available capacity of the corresponding beam after initial allocation; the set to be optimized includes the location and required capacity of unallocated users.

4. The satellite beam coverage adjustment method based on reinforcement learning according to claim 1, characterized in that, In the optimization adjustment, the adjustment actions performed by the agent include adjusting the beam center direction and beam width, and the reinforcement learning network is a DQN network.

5. The satellite beam coverage adjustment method based on reinforcement learning according to claim 1, characterized in that, In the optimization and adjustment, the service quality priority principle for selecting the appropriate beam for multi-beam coverage users includes: Select the beam that meets the capacity requirements and has the strongest signal; The user adjustment method for the supercapacity beam is as follows: users are removed from the beam in order of user priority from low to high until the total user demand capacity within the beam does not exceed the available capacity.

6. The satellite beam coverage adjustment method based on reinforcement learning according to claim 1, characterized in that, In the optimization and adjustment, the experience includes the agent's state before adjustment, the actions performed, the final reward that integrates local and global rewards, and the next state after adjustment; The method for updating network parameters based on the experience is as follows: sample batches of data from the experience to update the network parameters of the agent.

7. The satellite beam coverage adjustment method based on reinforcement learning according to claim 1, characterized in that, In the aforementioned optimization and adjustment, the rules for creating new beams are as follows: If the set to be optimized still contains unassigned users and the existing number of beams is less than the total number of beams, then a new beam is created with the position of the highest priority user among the unassigned users as the center point.

8. A satellite beam coverage adjustment system based on reinforcement learning, characterized in that, The system is used to implement the satellite beam coverage adjustment method based on reinforcement learning as described in any one of claims 1 to 7.

Images