Beam resource management method, device and electronic device

Abstract

The present invention provides a beam resource management method, device and electronic device, which relate to the technical field of satellite communications. The method uses the data flow matrix of the cell set served by the target satellite constellation in the current time slot as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the illumination direction or bandwidth allocation strategy of its corresponding beam in the current time slot according to the state under preset constraints, and then uses the illumination direction and bandwidth allocation strategy of all beams in the current time slot as the beam resource management strategy of the target satellite constellation. A beam corresponds to a pair of agents in the model, which are used for the illumination direction and bandwidth allocation strategy of the beam respectively, and the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, the total load difference and the beam interference intensity. Therefore, the method can effectively improve system performance and signal quality.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite communications, and in particular to a beam resource management method, device and electronic equipment. Background Art

[0002] For many years, satellite communication operators have been committed to providing on-orbit satellites with a certain degree of flexibility to adapt to the dynamically changing communication market needs. Beam Hopping (BH) technology is a key technology to achieve dynamic reconstruction of satellite resources and functions and improve the on-orbit efficiency and flexibility of payloads. It uses phased array antenna technology to control the spatial pointing of satellite-borne beams, and based on the ideas of time division multiplexing and frequency division multiplexing to achieve multiple access in the time domain and frequency, promote the efficient use of the "time-space-frequency" multi-domain resources of multi-beam satellites, and point the point beam from the established coverage layout to the desired area, thereby realizing the function of dynamically adjusting the receiving and transmitting coverage areas and beam positions.

[0003] Due to the complex space environment, time-varying network topology, payload limitations, uneven traffic distribution, and a surge in user tasks, beam resource management faces more severe internal and external disturbances and conflicts, and the challenge of system reliability needs to be solved urgently. The beam resource management method in the prior art has the following problems: 1. Under the condition of multi-user multi-task concurrency, the communication load of a single satellite is too heavy, and processing a large number of concurrent requests at the same time may cause communication delays, data loss, service interruptions, etc., reducing communication quality and user experience; 2. When multiple beams cover hot spots, strong interference between beams leads to poor signal quality, resulting in reduced data transmission rate, signal attenuation, increased delay, increased packet loss rate, etc., and after the signal quality decreases under interference conditions, it is necessary to retransmit data multiple times, which leads to network congestion and reduces the overall system performance. Summary of the invention

[0004] The object of the present invention is to provide a beam resource management method, device and electronic device to alleviate the technical problems of poor system performance and signal quality existing in the beam resource management method in the prior art.

[0005] In a first aspect, the present invention provides a beam resource management method, comprising: obtaining a data flow matrix of a cell set served by a target satellite constellation in a current time slot; wherein the data flow matrix is ​​constructed based on the data packet queues of all cells in the cell set in the current time slot; using the data flow matrix as the state of each agent in a target multi-agent deep reinforcement learning model, so that each agent outputs the irradiation direction or bandwidth allocation strategy of its corresponding beam in the current time slot according to the state under preset constraints; wherein each beam equipped by each satellite in the target satellite constellation corresponds to two agents in the target multi-agent deep reinforcement learning model, one agent is used to output the irradiation direction of its corresponding beam, and the other agent is used to output the bandwidth allocation strategy of its corresponding beam; the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity of the target satellite constellation; the preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; based on the irradiation direction and bandwidth allocation strategy of all beams in the current time slot, the beam resource management strategy of the target satellite constellation is determined.

[0006] Optionally, the calculation process of the total throughput of the target satellite constellation includes: obtaining the transmission power of each beam equipped by each satellite in the target satellite constellation and the channel coefficient of each visible cell in the current time slot; calculating the signal power of the target beam of the target satellite illuminating the target cell based on the transmission power of the target beam of the target satellite and the channel coefficient of the target cell; wherein the target satellite represents any satellite in the target satellite constellation, the target beam represents any beam of all beams equipped by the target satellite, and the target cell represents any cell of all visible cells of the target satellite; based on the bandwidth allocation strategy of all beams in the current time slot, calculating the first frequency overlapping factor between the target beam and the inter-satellite beam, and the second frequency overlapping factor between the target beam and the intra-satellite beam; based on the first frequency overlapping factor, the second frequency overlapping factor, and the current time slot, The interference power of the target cell illuminated by the target beam of the target satellite is calculated based on the illumination direction, transmission power and channel coefficient of all beams; the noise power spectral density and the sub-band bandwidth of the target satellite are obtained; based on the noise power spectral density, the bandwidth allocation strategy and the sub-band bandwidth of the target beam in the current time slot, the noise power of the target cell illuminated by the target beam of the target satellite is calculated; based on the signal power, interference power, noise power, the illumination direction of the target beam, the bandwidth allocation strategy and the sub-band bandwidth of the target satellite, the channel capacity of the target cell illuminated by the target beam in the current time slot is determined; based on the channel capacity and the data packet queue of the target cell in the current time slot, the throughput of the target cell illuminated by the target beam of the target satellite is determined; based on the illumination direction of all beams in the current time slot and the throughput of the cell illuminated by all beams, the total throughput of the target satellite constellation is determined.

[0007] Optionally, the calculation process of the total delay difference of the target satellite constellation includes: obtaining the waiting time of each data packet in the data packet queue of each visible cell of the target satellite in the current time slot; wherein the target satellite represents any satellite in the target satellite constellation; based on the waiting time of each data packet in the data packet queue of the target cell in the current time slot, calculating the average waiting time of the data packets of the target cell; wherein the target cell represents any cell among all visible cells of the target satellite; based on the maximum average waiting time of the data packets and the minimum average waiting time of the data packets in the visible cells of the target satellite, determining the delay difference of the visible cells of the target satellite; based on the illumination direction of all beams in the current time slot and the delay difference of the visible cells of all satellites in the target satellite constellation, determining the total delay difference of the target satellite constellation.

[0008] Optionally, the calculation process of the total load difference of the target satellite constellation includes: calculating the load of the target satellite in the current time slot based on the illumination direction of each beam equipped with the target satellite in the current time slot and the throughput of the illuminated cell; determining the total load difference of the target satellite constellation based on the maximum load and minimum load in the target satellite constellation in the current time slot.

[0009] Optionally, the calculation process of the beam interference intensity of the target satellite constellation includes: calculating the intra-satellite beam interference intensity of the target satellite based on the illumination directions of all beams equipped by the target satellite in the current time slot; calculating the inter-satellite beam interference intensity between the target satellite and other satellites based on the illumination directions of all beams in the target satellite constellation in the current time slot; calculating the inter-beam co-channel interference intensity of all adjacent visible cells of the target satellite based on the illumination directions of all beams in the current time slot and the bandwidth allocation strategy; determining the beam interference intensity of the target satellite in the current time slot based on the intra-satellite beam interference intensity, inter-satellite beam interference intensity and inter-beam co-channel interference intensity; and taking the sum of the beam interference intensities of all satellites in the target satellite constellation in the current time slot as the beam interference intensity of the target satellite constellation.

[0010] Optionally, the global reward calculation process includes: using the formula Calculate the global reward; where, represents the total throughput of the target satellite constellation, represents the total delay difference of the target satellite constellation, represents the total load difference of the target satellite constellation, represents the beam interference intensity of the target satellite s at the current time slot t, represents the total number of satellites in the target satellite constellation, represents the preset throughput limit, Indicates the preset delay difference limit. Indicates the preset load difference limit, represents the global reward of all agents in the current time slot t.

[0011] In a second aspect, the present invention provides a beam resource management device, including: an acquisition module, used to obtain a data flow matrix of a cell set served by a target satellite constellation in a current time slot; wherein the data flow matrix is ​​constructed based on the data packet queues of all cells in the cell set in the current time slot; a first determination module, used to use the data flow matrix as the state of each agent in a target multi-agent deep reinforcement learning model, so that each agent outputs the illumination direction or bandwidth allocation strategy of its corresponding beam in the current time slot according to the state under preset constraints; wherein each beam equipped by each satellite in the target satellite constellation corresponds to the target multi-agent deep reinforcement learning model. There are two intelligent agents in the reinforcement learning model, one agent is used to output the illumination direction of its corresponding beam, and the other agent is used to output the bandwidth allocation strategy of its corresponding beam; the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity of the target satellite constellation; the preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; the second determination module is used to determine the beam resource management strategy of the target satellite constellation based on the illumination direction and bandwidth allocation strategy of all beams in the current time slot.

[0012] In an optional implementation, it also includes: a first calculation module, used to calculate the total throughput of the target satellite constellation; the first calculation module is specifically used to: obtain the transmission power of each beam equipped by each satellite in the target satellite constellation and the channel coefficient of each visible cell in the current time slot; based on the transmission power of the target beam of the target satellite and the channel coefficient of the target cell, calculate the signal power of the target beam of the target satellite illuminating the target cell; wherein the target satellite represents any satellite in the target satellite constellation, the target beam represents any beam of all beams equipped by the target satellite, and the target cell represents any cell of all visible cells of the target satellite; based on the bandwidth allocation strategy of all beams in the current time slot, calculate the first frequency overlap factor between the target beam and the inter-satellite beam, and the second frequency overlap factor between the target beam and the intra-satellite beam; based on the first frequency overlap factor, the second The interference power of the target cell illuminated by the target beam of the target satellite is calculated based on the frequency overlap factor, the illumination direction of all beams in the current time slot, the transmission power and the channel coefficient; the noise power spectral density and the sub-band bandwidth of the target satellite are obtained; based on the noise power spectral density, the bandwidth allocation strategy and the sub-band bandwidth of the target beam in the current time slot, the noise power of the target cell illuminated by the target beam of the target satellite is calculated; based on the signal power, interference power, noise power, the illumination direction of the target beam, the bandwidth allocation strategy and the sub-band bandwidth of the target satellite, the channel capacity of the target cell illuminated by the target beam in the current time slot is determined; based on the channel capacity and the data packet queue of the target cell in the current time slot, the throughput of the target cell illuminated by the target beam of the target satellite is determined; based on the illumination direction of all beams in the current time slot and the throughput of the cell illuminated by all beams, the total throughput of the target satellite constellation is determined.

[0013] In a third aspect, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, it implements the beam resource management method of any one of the aforementioned implementation modes.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the beam resource management method of any one of the aforementioned implementation modes is implemented.

[0015] The present invention provides a beam resource management method, which uses the data flow matrix of the cell set served by the target satellite constellation in the current time slot as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the irradiation direction or bandwidth allocation strategy of the corresponding beam in the current time slot according to the state under the preset constraint conditions, and then uses the irradiation direction and bandwidth allocation strategy of all beams in the current time slot as the beam resource management strategy of the target satellite constellation. Among them, one beam corresponds to a pair of agents in the target multi-agent deep reinforcement learning model, which are used to output the irradiation direction and bandwidth allocation strategy of the beam respectively. The preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; and the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, the total load difference and the beam interference intensity. Therefore, the method can maximize the total throughput of the target satellite constellation, minimize the total delay difference, the total load difference and the beam interference intensity, thereby effectively improving the system performance and signal quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a beam resource management method provided by an embodiment of the present invention;

[0018] Figure 2 Schematic diagram of the target multi-agent deep reinforcement learning model in an embodiment of the present invention;

[0019] Figure 3 It is a schematic diagram of three types of interference considered when beam resource management is performed on a target satellite constellation according to an embodiment of the present invention;

[0020] Figure 4 A functional module diagram of a beam resource management device provided by an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] In the existing technology, under the condition of multi-user and multi-task concurrency, the communication load of a single satellite is too heavy, and processing a large number of concurrent requests at the same time may cause communication delays, data loss, service interruptions, etc., reducing communication quality and user experience. For example, in a maritime satellite communication system, the satellite may need to process the communication requests of thousands of ships distributed in a vast sea area at the same time. Each ship may have different communication needs, such as voice calls, data transmission, emergency calls, etc. When multiple ships try to establish a communication connection with the satellite at the same time or multiple high-bandwidth applications (such as high-definition video transmission) are running at the same time, the communication load of the satellite will increase rapidly. If the on-board processing power or bandwidth resources are not sufficient to cope with such high concurrency requirements, the load will be too heavy, resulting in a significant decline in user experience.

[0026] In addition, when multiple beams cover hotspots, strong interference between beams leads to poor signal quality, resulting in reduced data transmission rate, signal attenuation, increased delay, increased packet loss rate and other problems. For example, in urban hotspots with high user density and dense buildings, frequent service and simultaneous illumination of high-frequency bands (such as millimeter waves) may cause frequency overlap and signal aliasing, which causes users to encounter problems such as freezes, interruptions or delays when performing data transmission, video calls or online games, seriously affecting user experience. In addition, due to the degradation of signal quality under interference conditions, data needs to be retransmitted multiple times, resulting in network congestion and reducing overall system performance.

[0027] In view of this, an embodiment of the present invention provides a beam resource management method, which improves the active coordination capability and interference avoidance capability of beam resource management around the requirements of high concurrency and anti-interference. First, a theoretical model of a multi-satellite collaborative beam hopping system is built to characterize the system and channel status, reveal the multi-satellite beam illumination direction and resource allocation decision-making mechanism, derive key performance characterizations, and model multi-satellite collaborative optimization objectives. Secondly, considering the interference avoidance beam hopping strategy based on spatial isolation, modeling intra-satellite beam interference, inter-satellite beam interference, and adjacent cell beam interference indicators, and setting a penalty mechanism, using convex optimization theory to model the optimization target, and optimizing beam resource allocation and spatial pointing based on a multi-agent reinforcement learning algorithm.

[0028] After offline training of the initial multi-agent deep reinforcement learning model, the target multi-agent deep reinforcement learning model is obtained, and then the target multi-agent deep reinforcement learning model is deployed on a satellite with onboard processing capabilities or in a ground network operation control center, so that flexible, dynamic and accurate beam coverage and resource scheduling can be achieved according to ground traffic requirements. The beam resource management method provided in an embodiment of the present invention is described in detail below.

[0029] Embodiment 1

[0030] Figure 1 A flow chart of a beam resource management method provided by an embodiment of the present invention is as follows Figure 1 As shown, the method specifically comprises the following steps:

[0031] Step S102, obtaining a data flow matrix of a set of cells served by a target satellite constellation in a current time slot.

[0032] The data traffic matrix is ​​constructed based on the data packet queues of all cells in the cell set in the current time slot.

[0033] Specifically, the target satellite constellation is expressed as Satellites at orbital altitude H , Satellite Joint Services Residential area , where each satellite is equipped with beam Serve Visible cell , the visible cell of the satellite is also the cell covered by satellite communication. In the embodiment of the present invention, s, c, k and v represent the sth satellite, the cth cell, the kth beam and the vth visible cell respectively. Since the coverage of multiple satellites on some cells overlaps, and , that is, the union of the visible cells of all satellites in the target satellite constellation is all cells, where represents the visible cell set of satellite s, Indicates the number of visible cells of satellite s.

[0034] The data packets to be sent in each cell (that is, data traffic) form a data packet queue. When the beam illuminates a cell, some of the data packets at the head of the data packet queue of the cell will be transmitted, and new data packets will be generated in each time slot and added to the end of the queue. Data packets arriving at different time slots have different waiting times in the queue. From the end of the queue, the waiting time for the newly arrived data packet is 1 time slot, that is, from the end of the queue to the head of the queue, there are data packets waiting for 1 time slot, 2 time slots... T time slots. Among them, T represents the life cycle of the data packet. Data packets in the data packet queue that have waited for more than T time slots will be discarded.

[0035] Therefore, in the embodiment of the present invention, the data flow matrix of the cell set served by the target satellite constellation in the current time slot is: ; Can be abbreviated as .in, , represents the data traffic queue of cell c in time slot t, It means that cell c has been waiting for data packets for i time slots in time slot t. The total number of data packets in the cell data flow queue is the data flow value of the cell, that is, .

[0036] Based on this, the data flow of the visible cell of satellite s is expressed as: ;in, It means that the cell v covered by satellite s has been waiting for data packets for i time slots in time slot t. It should be noted that the data packet queue of each cell in each time slot has nothing to do with which satellite beam is irradiated by it. In other words, , Indicates that cell v has waited for the data packet of i time slots in time slot t, Indicates that the cell v covered by satellite s has been waiting for the data packet of i time slots at time slot t. It indicates that the cell v covered by the beam k of the satellite s has been waiting for the data packet for i time slots at time slot t.

[0037] Step S104, using the data flow matrix as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the illumination direction or bandwidth allocation strategy of its corresponding beam in the current time slot according to the state under preset constraints.

[0038] The embodiment of the present invention is a method for managing the beam resources of a target satellite constellation based on multi-agent deep reinforcement learning technology. The satellite observes the data flow requirements of each cell on the ground and constructs a data flow matrix as the input of the agent. All agents in the target multi-agent deep reinforcement learning model share the same global state to promote the collaborative optimization of multiple satellites and multiple beams. That is, the states of all agents are the data flow matrix of the cell set served by the target satellite constellation in the current time slot. , each agent outputs the corresponding action decision according to the state of each time slot, and the set of all agent action decisions is the beam resource management strategy of the target satellite constellation.

[0039] The embodiment of the present invention explores the multi-satellite coordinated beam resource management strategy around the downlink. Based on the phased array antenna and time division multiplexing technology, it can be found that the beam resource management of beam hopping can be defined as the task of inferring the beam illumination direction of each satellite in each time slot according to the ground traffic demand, which is referred to as beam allocation. Secondly, the bandwidth resources of the satellite directly affect the transmission capacity of the beam. Due to the limited onboard payload, reasonable, efficient and flexible bandwidth resource allocation is the key to improving the operating efficiency of the multi-satellite coordinated system.

[0040] Figure 2 is an architecture diagram of the target multi-agent deep reinforcement learning model in an embodiment of the present invention, such as Figure 2 As shown in Figure 2, each beam equipped by each satellite in the target satellite constellation corresponds to two agents in the target multi-agent deep reinforcement learning model, one of which is used to output the irradiation direction of its corresponding beam, that is, Figure 2 The beam allocation decision in is used to indicate the cell covered by the beam, and another agent is used to output the bandwidth allocation strategy of its corresponding beam, that is, Figure 2 The bandwidth allocation decision in is used to indicate the bandwidth reuse scheme of the beam. In other words, the target multi-agent deep reinforcement learning model includes For the agents, each pair of agents is responsible for optimizing the beam allocation decision and bandwidth allocation decision for one beam of a satellite.

[0041] Based on the above description, the state of the agent corresponding to the beam k of satellite s in time slot t is expressed as: ; The actions of the two agents corresponding to the beam k of satellite s in time slot t are expressed as: ,in, , represents the irradiation direction of beam k of satellite s at time slot t, represents the beam set of satellite s, is a binary variable, , It means that cell v within the coverage of satellite s is illuminated by beam k in time slot t. It means that cell v within the coverage of satellite s is not illuminated by beam k in time slot t. It represents the bandwidth allocation strategy of beam k of satellite s in time slot t.

[0042] In the embodiment of the present invention, when all agents perform action selection, in order to avoid wasting resources, the preset constraints include: ① The number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, that is, the number of satellites activated in each time slot is equal to the number of cells it serves in each time slot. beams and simultaneously serve Different cells are represented by: In the embodiment of the present invention, any time slot , that is, cells that are not within the visible range of satellite s Cannot be illuminated by its beam; ② Each cell is served by only one beam in each time slot, expressed as: .

[0043] The goal of an embodiment of the present invention is to maximize the throughput of the target satellite constellation, minimize the delay difference between ground cells, minimize the load difference of multiple satellites, and minimize the beam interference intensity of multiple satellites. Therefore, the global reward of all agents in the target multi-agent deep reinforcement learning model is set to be positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity of the target satellite constellation.

[0044] In the embodiment of the present invention, the target multi-agent deep reinforcement learning model is a model obtained by training the initial multi-agent deep reinforcement learning model with the global reward maximization as the optimization goal. Optionally, when training the initial multi-agent deep reinforcement learning model, the number of newly generated data packets in the cell v covered by the satellite s at the time slot t is It can be generated according to Poisson distribution and the traffic arrival rate of each cell can be set Different from each other to simulate the uneven distribution characteristics of ground users, the number of packets newly generated by cell v covered by satellite s in time slot t The probability is: ,in, It represents the traffic arrival rate of cell v covered by satellite s at time slot t.

[0045] In the initial multi-agent deep reinforcement learning model, all agents output actions according to the observed environmental states. The environment makes corresponding joint decisions according to the joint actions of all agents and gives global reward feedback from the environment. All agents share the same global reward to promote the collaborative optimization of multi-satellite and multi-beam. The network parameters of the agents are updated in combination with the preset neural network parameter update algorithm. The agents adjust the next decision direction according to the reward feedback: while performing interference avoidance, jointly optimize the throughput and load balancing of the multi-satellite collaborative system and the delay fairness of the ground cell. Repeat the above process to continuously optimize the model network parameters until the model reaches a convergence state, and obtain the target multi-agent deep reinforcement learning model.

[0046] Step S106, determining the beam resource management strategy of the target satellite constellation based on the illumination directions and bandwidth allocation strategies of all beams in the current time slot.

[0047] According to the description above, the illumination direction and bandwidth allocation strategy of all beams in the previous time slot is the beam resource management strategy of the target satellite constellation, which is expressed as: ,in, ; ; represents the beam allocation pattern of satellite s in time slot t, represents the irradiation direction of the beam k of satellite s in time slot t, according to It can determine which cells are illuminated by the beam in the visible coverage area of ​​each satellite in time slot t; represents the bandwidth allocation pattern of satellite s in time slot t, It represents the bandwidth allocation strategy of beam k of satellite s in time slot t.

[0048] In the embodiment of the present invention, the bandwidth allocation of the beam is based on the frequency division multiplexing technology, which divides the total bandwidth resources of each satellite into multiple sub-bands. Different beams can select the optimal bandwidth scheme required from a variety of different multiplexing schemes, flexibly allocate spectrum resources, and improve spectrum utilization. If the total bandwidth resources of satellite s are , the number of sub-bands is , then the bandwidth of each sub-band is , so the total number of bandwidth reuse schemes is: Therefore, (that is, to limit the optional range of the bandwidth reuse scheme of the beam), (That is, the bandwidth occupied by beam k of satellite s does not exceed the total bandwidth), ,in, Represents the number of sub-bands occupied by the bandwidth allocation strategy selected by satellite s and beam k in time slot t.

[0049] For ease of understanding, the following example illustrates that if the number of satellite s's sub-bands is 5, and the 5 sub-bands are {a, b, c, d, e}, then there are 15 optional bandwidth schemes for the beam: ① Occupy 1 sub-band: a, b, c, d, e; ② Occupy 2 sub-bands: ab, bc, cd, de; ③ Occupy 3 sub-bands: abc, bcd, cde; ④ Occupy 4 sub-bands abcd, bcde; ⑤ Occupy 5 sub-bands abcde. If the bandwidth allocation strategy selected by satellite s beam k in time slot t is ab, then =2.

[0050] The embodiment of the present invention provides a beam resource management method, which uses the data flow matrix of the cell set served by the target satellite constellation in the current time slot as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the irradiation direction or bandwidth allocation strategy of the corresponding beam in the current time slot according to the state under the preset constraint conditions, and then uses the irradiation direction and bandwidth allocation strategy of all beams in the current time slot as the beam resource management strategy of the target satellite constellation. Among them, one beam corresponds to a pair of agents in the target multi-agent deep reinforcement learning model, which are used to output the irradiation direction and bandwidth allocation strategy of the beam respectively. The preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; and the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity. Therefore, the method can maximize the total throughput of the target satellite constellation, minimize the total delay difference, total load difference and beam interference intensity, thereby effectively improving system performance and signal quality.

[0051] In an embodiment of the present invention, the calculation process of the total throughput of the target satellite constellation includes the following steps:

[0052] Step S201, obtaining the transmission power of each beam equipped by each satellite in the target satellite constellation and the channel coefficient illuminating each visible cell in the current time slot.

[0053] Step S202, based on the transmission power of the target beam of the target satellite and the channel coefficient of the target cell, calculate the signal power of the target beam of the target satellite illuminating the target cell; wherein the target satellite represents any satellite in the target satellite constellation, the target beam represents any beam of all beams equipped with the target satellite, and the target cell represents any cell of all visible cells of the target satellite.

[0054] Specifically, the signal power of satellite s beam k irradiating cell v in time slot t is expressed as: ;in, represents the transmission power of satellite s beam k, , represents the maximum transmit power of the beam, , Represents the total power of the satellite, that is, the sum of the transmit powers of all beams of the satellite does not exceed the total power of the satellite; Represents the channel coefficient of satellite s beam k illuminating cell v in time slot t.

[0055] Generally, , represents the transmitting antenna gain of satellite s, represents the free space signal propagation path loss, represents the receiving antenna gain of the visible cell v. According to ITU standard ITU-RS.672-4, when the beam k of satellite s illuminates the visible cell v, the off-axis angle The transmit antenna gain at is defined as follows: ; represents the near sidelobe level required for peak gain, a and b are preset constants, and the corresponding relationship between the three values ​​is shown in the following Table 1; Indicates 3dB wave width off-axis angle half of; It represents the off-axis angle when the transmission gain is 0 dBi; represents the maximum transmit antenna gain on the main lobe and is calculated as: .

[0056] Table 1 According to ITU-R S.672-4 standard, a, b and The corresponding relationship

[0057]

[0058] Free space signal propagation path loss , represents the satellite orbit altitude, Indicates the satellite downlink frequency.

[0059] If the ground cell uses an omnidirectional antenna deployed on the ground station, the receiving antenna gain of the visible cell v can be derived according to the following formula: ; Represents the wavelength of the communication signal between the satellite and the ground station; represents the effective aperture area of ​​the antenna, and D represents the effective aperture diameter.

[0060] Step S203: Based on the bandwidth allocation strategies of all beams in the current time slot, a first frequency overlap factor between the target beam and the inter-satellite beam and a second frequency overlap factor between the target beam and the intra-satellite beam are calculated.

[0061] The frequency overlap factor characterizes the degree of co-frequency interference between frequencies. The formula for the first frequency overlap factor of beam k and the intersatellite beam (beam equipped by other satellites) is: The formula for the second frequency overlap factor of beam k and the intra-satellite beam (other beams equipped by the satellite to which the target beam belongs) is: ;in, indicates a satellite different from satellite s; Indicates satellite A beam different from beam k; Indicates satellite A cell different from the visible cell v; represents the beam in satellite s that is different from beam k; Indicates the cells in satellite s that are different from the visible cell v.

[0062] Step S204, based on the first frequency overlapping factor, the second frequency overlapping factor, the irradiation directions of all beams in the current time slot, the transmission power and the channel coefficient, calculate the interference power of the target beam of the target satellite irradiating the target cell.

[0063] In the embodiment of the present invention, the interference power of the target beam of the target satellite irradiating the target cell is is the inter-satellite beam co-frequency interference power The co-frequency interference power of the satellite beam The sum of , , Based on the above formula, after determining the illumination direction, transmission power and channel coefficient of all beams in the current time slot and calculating the first frequency overlapping factor and the second frequency overlapping factor, the interference power of the target beam of the target satellite irradiating the target cell can be further calculated according to the above formula.

[0064] Step S205: Acquire the noise power spectrum density and the sub-band bandwidth of the target satellite.

[0065] Step S206, based on the noise power spectrum density, the bandwidth allocation strategy of the target beam in the current time slot and the sub-band bandwidth, the noise power of the target beam of the target satellite irradiating the target cell is calculated.

[0066] Specifically, the noise power of satellite s beam k irradiating cell v in time slot t is calculated as: ;in, represents the noise power spectral density, represents the bandwidth occupied by satellite s beam k in time slot t, , represents the number of sub-bands occupied by the bandwidth allocation strategy of beam k of satellite s in time slot t, represents the sub-band bandwidth of satellite s.

[0067] Step S207, based on the signal power, interference power, noise power, the illumination direction of the target beam, the bandwidth allocation strategy and the sub-band bandwidth of the target satellite, determine the channel capacity of the target beam illuminating the target cell in the current time slot.

[0068] After determining the signal power of the target satellite's target beam illuminating the target cell , interference power , Noise Power After that, the channel situation can be described. The signal-to-noise ratio of satellite s beam k irradiating cell v in time slot t can be expressed as: Based on the above channel situation information, the channel capacity characteristic model can be constructed to predict and evaluate the transmission capacity of the channel under different conditions, providing a basis for subsequent decision-making. Based on the Shannon formula, the channel capacity of satellite s beam k irradiating cell v in time slot t is modeled as: .

[0069] Step S208, based on the channel capacity and the data packet queue of the target cell in the current time slot, determine the throughput of the target satellite's target beam illuminating the target cell.

[0070] Step S209: determining the total throughput of the target satellite constellation based on the illumination directions of all beams in the current time slot and the throughput of the cells illuminated by all beams.

[0071] In the embodiment of the present invention, the throughput of the beam k of the satellite s illuminating the visible cell v in the time slot t is: Based on this, the total throughput of the target satellite constellation is calculated as: .

[0072] In an embodiment of the present invention, the calculation process of the total delay difference of the target satellite constellation includes the following steps:

[0073] Step S301, obtaining the waiting time of each data packet in the data packet queue of the current time slot of each visible cell of the target satellite; wherein the target satellite represents any satellite in the target satellite constellation.

[0074] Step S302, based on the waiting time of each data packet in the data packet queue of the target cell in the current time slot, the average waiting time of the data packets of the target cell is calculated; wherein the target cell represents any cell in all visible cells of the target satellite.

[0075] Based on the above description, it can be seen that the visible cell v of satellite s has a packet in the packet queue of time slot t. The waiting time is i (unit: time slot), so the average waiting time of the data packet of the visible cell v of satellite s in time slot t is calculated as: .

[0076] Step S303 : determining the delay difference of the visible cell of the target satellite based on the maximum average waiting time of data packets and the minimum average waiting time of data packets in the visible cell of the target satellite.

[0077] That is, the formula for the delay difference of satellite s visible cell v in time slot t is: Based on this, the delay difference of all satellites in the target satellite constellation in time slot t can be modeled as: Based on the algorithm for the delay difference in the above text, it can be seen that the elements in each row of the matrix have the same value.

[0078] Step S304: determine the total delay difference of the target satellite constellation based on the illumination directions of all beams in the current time slot and the delay difference of visible cells of all satellites in the target satellite constellation.

[0079] In the embodiment of the present invention, the total delay difference of the target satellite constellation is calculated as: ,in, , that is, .

[0080] In an embodiment of the present invention, the calculation process of the total load difference of the target satellite constellation includes the following steps:

[0081] Step S401, calculating the load of the target satellite in the current time slot based on the illumination direction of each beam equipped by the target satellite in the current time slot and the throughput of the illuminated cell.

[0082] Step S402: determining a total load difference of the target satellite constellation based on a maximum load and a minimum load in the target satellite constellation in the current time slot.

[0083] The embodiment of the present invention exerts the system joint efficiency through load balancing, and defines the load of the satellite s in the t time slot as the sum of the throughputs of the cells it serves. Therefore, after determining the irradiation direction of each beam equipped by the satellite s in the t time slot and the throughput of the irradiated cell, the load of the satellite s in the t time slot can be calculated: , and set the load of each satellite in the initial state to 0, that is, Based on this, the load of all satellites in time slot t can be modeled as: The total load difference of the target satellite constellation is expressed as: .

[0084] Figure 3The schematic diagram of the three types of interference considered when the embodiment of the present invention performs beam resource management on the target satellite constellation is shown. Specifically, intra-satellite beam interference: if different beams of the same satellite repeatedly illuminate the same cell, it will cause frequency overlap, signal superposition and other phenomena, thereby causing intra-satellite beam interference and affecting communication quality; similarly, inter-satellite beam interference: within the overlapping coverage of inter-satellite, different beams of different satellites repeatedly illuminate the same cell, which will also cause inter-satellite beam interference and affect communication quality; in addition, adjacent cell beam interference: if the irradiation beams of adjacent cells contain the same frequency band, it will also cause co-frequency interference between beams, directly affecting the signal-to-noise ratio and channel capacity of the wireless channel, thereby affecting the transmission performance of the system. Therefore, in order to avoid problems such as beam interference and decreased communication quality, the embodiment of the present invention models the above three interference scenarios as penalty terms as part of the reinforcement learning reward function, and guides or limits the optimization direction of the subsequent reinforcement learning agent action through the penalty mechanism to accurately control the spatial pointing of the beam.

[0085] The embodiment of the present invention establishes a beam interference penalty quantitative index model to measure the beam interference intensity in an actual system, wherein the calculation process of the beam interference intensity of the target satellite constellation includes the following steps:

[0086] Step S501, based on the irradiation directions of all beams equipped by the target satellite in the current time slot, the intra-satellite beam interference intensity of the target satellite is calculated.

[0087] Based on the meaning of intra-satellite beam interference, each beam in the satellite should only be responsible for a specific cell in the visible area to avoid interference. Therefore, the intra-satellite beam interference intensity of satellite s in time slot t is The formula is: , ,in, represents the preset penalty coefficient, optionally, The value is 0.01. The penalty amount is multiplied by the number of repeated exposures.

[0088] Step S502, based on the illumination directions of all beams in the target satellite constellation in the current time slot, calculate the inter-satellite beam interference intensity between the target satellite and other satellites.

[0089] Based on the meaning of inter-satellite beam interference, it can be known that within the overlapping coverage of satellites, each cell can only be illuminated by one beam. Therefore, the inter-satellite beam interference intensity between satellite s and the rest of the satellites in time slot t is The formula is: , .

[0090] Step S503, based on the illumination directions of all beams in the current time slot and the bandwidth allocation strategy, the co-channel interference intensity between beams of all adjacent visible cells of the target satellite is calculated.

[0091] Specifically, spectrum utilization must be reasonably planned to avoid adjacent cells containing the same frequency band, thereby reducing the possibility of interference. Therefore, the formula for the co-channel interference intensity between beams of all adjacent visible cells of satellite s is: , , , Represents the set of neighboring cells of cell v.

[0092] Step S504: Determine the beam interference intensity of the target satellite in the current time slot based on the intra-satellite beam interference intensity, the inter-satellite beam interference intensity and the inter-beam co-channel interference intensity.

[0093] Step S505: The sum of the beam interference intensities of all satellites in the target satellite constellation in the current time slot is used as the beam interference intensity of the target satellite constellation.

[0094] In the embodiment of the present invention, the formula for calculating the beam interference intensity of satellite s in time slot t is: ; Therefore, the beam interference strength of the target satellite constellation is .

[0095] The present invention is implemented to comprehensively reflect the system performance requirements while comprehensively considering interference avoidance, including: maximizing system throughput, minimizing delay difference and load difference, and optimizing beam layout by accurately controlling the spatial pointing of beams, effectively avoiding inter-beam interference in hotspot-dense areas. Therefore, based on convex optimization technology, the model optimization problem is modeled as a solvable multi-objective, multi-constrained mathematical model, and the optimal solution that meets the preset constraints is found by solving the model, and this optimal solution is a trade-off between multiple objectives to achieve overall optimization of system performance.

[0096] In view of this, in an embodiment of the present invention, the calculation process of the global reward includes: Using the formula Calculate the global reward; where, represents the total throughput of the target satellite constellation, represents the total delay difference of the target satellite constellation, represents the total load difference of the target satellite constellation, represents the beam interference intensity of the target satellite s at the current time slot t, represents the total number of satellites in the target satellite constellation, represents the preset throughput limit, Indicates the preset delay difference limit. Indicates the preset load difference limit. The above three limits can be the maximum throughput, delay difference and load difference in the batch data during the model training process. represents the global reward of all agents in the current time slot t. That is, in the model training phase, the optimization goal is: .

[0097] In summary, the embodiment of the present invention discloses the characterization and optimization mechanism of multi-satellite collaborative beam resources. First, a theoretical model of a multi-satellite collaborative beam hopping system is established. Aiming at multi-user, multi-task and high-concurrency scenarios, a beam resource management framework under multi-satellite collaboration is constructed and improved to achieve system joint efficiency and load balancing; secondly, the wireless channel environment modeling is improved, the wireless channel characteristics under multi-satellite collaboration are studied in depth, and an accurate channel environment model is established to provide theoretical support for the derivation and modeling of key metrics such as system transmission; finally, key evaluation indicators are derived, and key performance indicators in multi-satellite collaborative systems are sorted out, including throughput, service quality (corresponding to the "delay difference" above), load balancing, etc., to provide a quantitative basis for system optimization. In addition, the embodiment of the present invention also optimizes the interference avoidance beam technology under spatial isolation conditions to effectively avoid inter-beam interference. By accurately controlling the spatial pointing of the beam and optimizing the beam layout, inter-beam interference in hotspot-dense areas is effectively avoided. Through interference avoidance technology, the stable operation of the system is ensured, the signal quality is improved, and the stability and efficiency of data transmission are guaranteed.

[0098] Embodiment 2

[0099] An embodiment of the present invention further provides a beam resource management device, which is mainly used to execute the beam resource management method provided in the above-mentioned embodiment 1. The beam resource management device provided in the embodiment of the present invention is specifically introduced below.

[0100] Figure 4 A functional module diagram of a beam resource management device provided by an embodiment of the present invention, such as Figure 4 As shown, the device mainly includes: an acquisition module 10, a first determination module 20, and a second determination module 30, wherein:

[0101] The acquisition module 10 is used to obtain the data flow matrix of the cell set served by the target satellite constellation in the current time slot; wherein the data flow matrix is ​​constructed based on the data packet queues of all cells in the cell set in the current time slot.

[0102] The first determination module 20 is used to use the data flow matrix as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the illumination direction or bandwidth allocation strategy of the corresponding beam in the current time slot according to the state under preset constraints.

[0103] Among them, each beam equipped by each satellite in the target satellite constellation corresponds to two agents in the target multi-agent deep reinforcement learning model, one agent is used to output the illumination direction of its corresponding beam, and the other agent is used to output the bandwidth allocation strategy of its corresponding beam; the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity of the target satellite constellation; the preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot.

[0104] The second determination module 30 is used to determine the beam resource management strategy of the target satellite constellation based on the illumination directions and bandwidth allocation strategies of all beams in the current time slot.

[0105] The embodiment of the present invention provides a beam resource management device, which uses the data flow matrix of the cell set served by the target satellite constellation in the current time slot as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the irradiation direction or bandwidth allocation strategy of the corresponding beam in the current time slot according to the state under the preset constraint conditions, and then uses the irradiation direction and bandwidth allocation strategy of all beams in the current time slot as the beam resource management strategy of the target satellite constellation. Among them, one beam corresponds to a pair of agents in the target multi-agent deep reinforcement learning model, which are used to output the irradiation direction and bandwidth allocation strategy of the beam respectively. The preset constraint conditions include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; and the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity. Therefore, the device can maximize the total throughput of the target satellite constellation, minimize the total delay difference, total load difference and beam interference intensity, thereby effectively improving system performance and signal quality.

[0106] Optionally, the device further includes: a first calculation module, configured to calculate a total throughput of a target satellite constellation; the first calculation module is specifically configured to:

[0107] Obtain the transmission power of each beam equipped by each satellite in the target satellite constellation in the current time slot and the channel coefficient illuminating each visible cell.

[0108] Based on the transmission power of the target beam of the target satellite and the channel coefficient of the illuminated target cell, the signal power of the target beam of the target satellite illuminating the target cell is calculated; wherein the target satellite represents any satellite in the target satellite constellation, the target beam represents any beam of all beams equipped with the target satellite, and the target cell represents any cell of all visible cells of the target satellite.

[0109] Based on the bandwidth allocation strategy of all beams in the current time slot, the first frequency overlapping factor between the target beam and the inter-satellite beam and the second frequency overlapping factor between the target beam and the intra-satellite beam are calculated.

[0110] Based on the first frequency overlapping factor, the second frequency overlapping factor, the irradiation directions of all beams in the current time slot, the transmission power and the channel coefficient, the interference power of the target beam of the target satellite irradiating the target cell is calculated.

[0111] Get the noise power spectral density and sub-band bandwidth of the target satellite.

[0112] The noise power of the target beam of the target satellite irradiating the target cell is calculated based on the noise power spectral density, the bandwidth allocation strategy of the target beam in the current time slot, and the sub-band bandwidth.

[0113] Based on the signal power, interference power, noise power, the illumination direction of the target beam, the bandwidth allocation strategy and the sub-band bandwidth of the target satellite, the channel capacity of the target beam illuminating the target cell in the current time slot is determined.

[0114] Based on the channel capacity and the data packet queue of the target cell in the current time slot, the throughput of the target cell illuminated by the target beam of the target satellite is determined.

[0115] Based on the illumination directions of all beams in the current time slot and the throughput of the cells illuminated by all beams, the total throughput of the target satellite constellation is determined.

[0116] Optionally, the device further includes: a second calculation module, configured to calculate a total delay difference of the target satellite constellation; the second calculation module is specifically configured to:

[0117] The waiting time of each data packet in the data packet queue of the current time slot of each visible cell of the target satellite is obtained; wherein the target satellite represents any satellite in the target satellite constellation.

[0118] Based on the waiting time of each data packet in the data packet queue of the target cell in the current time slot, the average waiting time of the data packets of the target cell is calculated; wherein the target cell represents any cell in all visible cells of the target satellite.

[0119] The delay difference of the visible cell of the target satellite is determined based on the maximum average waiting time of the data packets and the minimum average waiting time of the data packets in the visible cell of the target satellite.

[0120] Based on the illumination directions of all beams in the current time slot and the delay differences of visible cells of all satellites in the target satellite constellation, the total delay difference of the target satellite constellation is determined.

[0121] Optionally, the device further includes: a third calculation module, configured to calculate a total load difference of the target satellite constellation; the third calculation module is specifically configured to:

[0122] Based on the illumination direction of each beam equipped by the target satellite in the current time slot and the throughput of the illuminated cell, the load of the target satellite in the current time slot is calculated.

[0123] Based on the maximum load and the minimum load in the target satellite constellation in the current time slot, a total load difference of the target satellite constellation is determined.

[0124] Optionally, the device further includes: a fourth calculation module, configured to calculate the beam interference intensity of the target satellite constellation; the fourth calculation module is specifically configured to:

[0125] Based on the illumination directions of all beams equipped by the target satellite in the current time slot, the intra-satellite beam interference intensity of the target satellite is calculated.

[0126] Based on the illumination directions of all beams in the target satellite constellation in the current time slot, the inter-satellite beam interference intensity between the target satellite and the remaining satellites is calculated.

[0127] Based on the illumination directions of all beams in the current time slot and the bandwidth allocation strategy, the co-channel interference intensity between beams of all adjacent visible cells of the target satellite is calculated.

[0128] Based on the intra-satellite beam interference intensity, the inter-satellite beam interference intensity and the inter-beam co-frequency interference intensity, the beam interference intensity of the target satellite in the current time slot is determined.

[0129] The sum of the beam interference intensities of all satellites in the target satellite constellation in the current time slot is taken as the beam interference intensity of the target satellite constellation.

[0130] Optionally, the device further includes: a fifth calculation module, configured to calculate a global reward; the fifth calculation module is specifically configured to:

[0131] Using the formula Calculate the global reward; where, represents the total throughput of the target satellite constellation, represents the total delay difference of the target satellite constellation, represents the total load difference of the target satellite constellation, represents the beam interference intensity of the target satellite s at the current time slot t, represents the total number of satellites in the target satellite constellation, represents the preset throughput limit, Indicates the preset delay difference limit. Indicates the preset load difference limit, represents the global reward of all agents in the current time slot t.

[0132] Embodiment 3

[0133] See also Figure 5 An embodiment of the present invention provides an electronic device, which includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0134] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0135] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0136] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0137] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0138] A computer program product of a beam resource management method, device, and electronic device provided in an embodiment of the present invention includes a computer-readable storage medium storing a non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0139] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0141] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0142] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0143] In addition, the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0144] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A beam resource management method, characterized in that: include: Obtaining a data flow matrix of a cell set served by a target satellite constellation in a current time slot; wherein the data flow matrix is ​​constructed based on data packet queues of all cells in the cell set in the current time slot; The data flow matrix is ​​used as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the illumination direction or bandwidth allocation strategy of the corresponding beam in the current time slot according to the state under preset constraints; Among them, each beam equipped by each satellite in the target satellite constellation corresponds to two agents in the target multi-agent deep reinforcement learning model, one agent is used to output the irradiation direction of its corresponding beam, and the other agent is used to output the bandwidth allocation strategy of its corresponding beam; the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity of the target satellite constellation; the preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; Determine the beam resource management strategy of the target satellite constellation based on the illumination directions and bandwidth allocation strategies of all beams in the current time slot; The calculation process of the beam interference strength of the target satellite constellation includes: Calculate the intra-satellite beam interference intensity of the target satellite based on the illumination directions of all beams equipped by the target satellite in the current time slot; Calculating the inter-satellite beam interference intensity between the target satellite and other satellites based on the illumination directions of all beams in the target satellite constellation in the current time slot; Based on the illumination directions of all beams in the current time slot and the bandwidth allocation strategy, the co-channel interference intensity between beams of all adjacent visible cells of the target satellite is calculated; Determine the beam interference intensity of the target satellite in the current time slot based on the intra-satellite beam interference intensity, the inter-satellite beam interference intensity and the inter-beam co-channel interference intensity; The sum of the beam interference intensities of all satellites in the target satellite constellation in the current time slot is used as the beam interference intensity of the target satellite constellation.

2. The beam resource management method according to claim 1, characterized in that: The calculation process of the total throughput of the target satellite constellation includes: Obtaining the transmission power of each beam equipped by each satellite in the target satellite constellation and the channel coefficient illuminating each visible cell in the current time slot; Based on the transmission power of the target beam of the target satellite and the channel coefficient of the target cell, the signal power of the target beam of the target satellite illuminating the target cell is calculated; wherein the target satellite represents any satellite in the target satellite constellation, the target beam represents any beam among all beams equipped by the target satellite, and the target cell represents any cell among all visible cells of the target satellite; Based on the bandwidth allocation strategy of all beams in the current time slot, a first frequency overlapping factor between the target beam and the inter-satellite beam and a second frequency overlapping factor between the target beam and the intra-satellite beam are calculated; Calculate the interference power of the target beam of the target satellite irradiating the target cell based on the first frequency overlapping factor, the second frequency overlapping factor, the irradiation directions of all beams in the current time slot, the transmit power and the channel coefficient; Acquire noise power spectral density and sub-band bandwidth of the target satellite; Calculate the noise power of the target beam of the target satellite irradiating the target cell based on the noise power spectrum density, the bandwidth allocation strategy of the target beam in the current time slot, and the sub-band bandwidth; Determine the channel capacity of the target cell illuminated by the target beam in the current time slot based on the signal power, the interference power, the noise power, the illumination direction of the target beam, the bandwidth allocation strategy and the sub-band bandwidth of the target satellite; Determine the throughput of the target cell illuminated by the target beam of the target satellite based on the channel capacity and the data packet queue of the target cell in the current time slot; Based on the illumination directions of all beams in the current time slot and the throughputs of the cells illuminated by all beams, the total throughput of the target satellite constellation is determined.

3. The beam resource management method according to claim 1, characterized in that: The calculation process of the total delay difference of the target satellite constellation includes: Obtaining the waiting time of each data packet in the data packet queue of the current time slot for each visible cell of the target satellite; wherein the target satellite represents any satellite in the target satellite constellation; Based on the waiting time of each data packet in the data packet queue of the target cell in the current time slot, the average waiting time of the data packets of the target cell is calculated; wherein the target cell represents any cell among all visible cells of the target satellite; Determine the delay difference of the visible cell of the target satellite based on the maximum average waiting time of data packets and the minimum average waiting time of data packets in the visible cell of the target satellite; Based on the illumination directions of all beams in the current time slot and the delay differences of visible cells of all satellites in the target satellite constellation, the total delay difference of the target satellite constellation is determined.

4. The beam resource management method according to claim 2, characterized in that: The calculation process of the total load difference of the target satellite constellation includes: Calculate the load of the target satellite in the current time slot based on the illumination direction of each beam equipped by the target satellite in the current time slot and the throughput of the illuminated cell; Based on the maximum load and the minimum load in the target satellite constellation in the current time slot, a total load difference of the target satellite constellation is determined.

5. The beam resource management method according to claim 1, characterized in that: The calculation process of the global reward includes: Using the formula Calculate the global reward; wherein, represents the total throughput of the target satellite constellation, represents the total delay difference of the target satellite constellation, represents the total load difference of the target satellite constellation, represents the beam interference intensity of the target satellite s at the current time slot t, represents the total number of satellites in the target satellite constellation, represents the preset throughput limit, Indicates the preset delay difference limit. Indicates the preset load difference limit, represents the global reward of all agents in the current time slot t.

6. A beam resource management device, characterized in that: include: An acquisition module, used to acquire a data flow matrix of a cell set served by a target satellite constellation in a current time slot; wherein the data flow matrix is ​​constructed based on data packet queues of all cells in the cell set in the current time slot; A first determination module is used to use the data flow matrix as the state of each agent in the target multi-agent deep reinforcement learning model, so that each agent outputs the illumination direction or bandwidth allocation strategy of the corresponding beam in the current time slot according to the state under preset constraints; Among them, each beam equipped by each satellite in the target satellite constellation corresponds to two agents in the target multi-agent deep reinforcement learning model, one agent is used to output the irradiation direction of its corresponding beam, and the other agent is used to output the bandwidth allocation strategy of its corresponding beam; the global reward of all agents is positively correlated with the total throughput of the target satellite constellation, and negatively correlated with the total delay difference, total load difference and beam interference intensity of the target satellite constellation; the preset constraints include: the number of beams equipped by each satellite is equal to the number of cells it serves in each time slot, and each cell is served by only one beam in each time slot; A second determination module is used to determine the beam resource management strategy of the target satellite constellation based on the illumination directions and bandwidth allocation strategies of all beams in the current time slot; The device also includes: a fourth calculation module, which is used to calculate the beam interference intensity of the target satellite constellation; the fourth calculation module is specifically used to: Calculate the intra-satellite beam interference intensity of the target satellite based on the illumination directions of all beams equipped by the target satellite in the current time slot; Calculating the inter-satellite beam interference intensity between the target satellite and other satellites based on the illumination directions of all beams in the target satellite constellation in the current time slot; Based on the illumination directions of all beams in the current time slot and the bandwidth allocation strategy, the co-channel interference intensity between beams of all adjacent visible cells of the target satellite is calculated; Determine the beam interference intensity of the target satellite in the current time slot based on the intra-satellite beam interference intensity, the inter-satellite beam interference intensity and the inter-beam co-channel interference intensity; The sum of the beam interference intensities of all satellites in the target satellite constellation in the current time slot is used as the beam interference intensity of the target satellite constellation.

7. The beam resource management device according to claim 6, characterized in that: Also includes: A first calculation module, configured to calculate a total throughput of the target satellite constellation; The first calculation module is specifically used for: Obtaining the transmission power of each beam equipped by each satellite in the target satellite constellation and the channel coefficient illuminating each visible cell in the current time slot; Based on the transmission power of the target beam of the target satellite and the channel coefficient of the target cell, the signal power of the target beam of the target satellite illuminating the target cell is calculated; wherein the target satellite represents any satellite in the target satellite constellation, the target beam represents any beam among all beams equipped by the target satellite, and the target cell represents any cell among all visible cells of the target satellite; Based on the bandwidth allocation strategy of all beams in the current time slot, a first frequency overlapping factor between the target beam and the inter-satellite beam and a second frequency overlapping factor between the target beam and the intra-satellite beam are calculated; Calculate the interference power of the target beam of the target satellite irradiating the target cell based on the first frequency overlapping factor, the second frequency overlapping factor, the irradiation directions of all beams in the current time slot, the transmit power and the channel coefficient; Acquire noise power spectral density and sub-band bandwidth of the target satellite; Calculate the noise power of the target beam of the target satellite irradiating the target cell based on the noise power spectrum density, the bandwidth allocation strategy of the target beam in the current time slot, and the sub-band bandwidth; Determine the channel capacity of the target cell illuminated by the target beam in the current time slot based on the signal power, the interference power, the noise power, the illumination direction of the target beam, the bandwidth allocation strategy and the sub-band bandwidth of the target satellite; Determine the throughput of the target cell illuminated by the target beam of the target satellite based on the channel capacity and the data packet queue of the target cell in the current time slot; Based on the illumination directions of all beams in the current time slot and the throughputs of the cells illuminated by all beams, the total throughput of the target satellite constellation is determined.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the beam resource management method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the beam resource management method according to any one of claims 1 to 5 is implemented.

Images