QoE-Driven Phased Array Beamforming and Beam Hopping Methods and Devices for Low Earth Orbit Satellites

Abstract

The present invention provides a phased array beamforming and beam hopping method and device for low-earth orbit satellites driven by QoE, which relates to the field of communication technologies and includes: determining the priority of resource requirements corresponding to each service to be transmitted at the current moment; performing beamforming based on the priority of resource requirements to determine the beams that can cover the target user terminals and the resource requirement degrees corresponding to the beams at the current moment; through the resource allocation model corresponding to the beams, according to the resource requirement degrees corresponding to the beams, determining the target resource allocation results corresponding to the beams at the current moment, so as to transmit the services to be transmitted uploaded by the target user terminals according to the target resource allocation results; wherein, the resource allocation model is dynamically maintained based on the target resource allocation results corresponding to the beams at historical moments. The present invention can effectively save the resource waste caused by unnecessary coverage of areas without users, and significantly improve the resource allocation performance of the downlink of low-earth orbit satellites.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular, to a phased array beamforming and hopping beam method and apparatus for low-earth orbit satellites driven by QoE. Background Art

[0002] With the increasing demand for mobile data communication in recent years, traditional satellite communication systems are difficult to meet the growing requirements for communication quality and transmission speed due to fixed beam coverage and limited spectrum resources. Currently, low-earth orbit satellites can utilize hopping beam antenna technology to significantly improve resource utilization efficiency and service flexibility by allocating resource blocks to multiple spot beams. However, due to the uneven geographical distribution of ground users and the large temporal differences in the traffic arrival density of different users, traditional hopping beam methods will cause resource waste due to unnecessary coverage of areas without users, thereby affecting resource utilization. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a phased array beamforming and hopping beam method and apparatus for low-earth orbit satellites driven by QoE, which can effectively save the resource waste caused by unnecessary coverage of areas without users and significantly improve the downlink resource allocation performance of low-earth orbit satellites.

[0004] In a first aspect, the present invention provides a phased array beamforming and hopping beam method for low-earth orbit satellites driven by QoE, including:

[0005] Receiving services to be transmitted uploaded by multiple user terminals, and determining the resource demand priority corresponding to each service to be transmitted at the current moment;

[0006] Performing beamforming based on the resource demand priority corresponding to each service to be transmitted to determine the beam that can cover the target user terminal and the resource demand degree corresponding to the beam at the current moment;

[0007] Through the resource allocation model corresponding to the beam, according to the resource demand degree corresponding to the beam, determining the target resource allocation result corresponding to the beam at the current moment, so as to transmit the service to be transmitted uploaded by the target user terminal according to the target resource allocation result; wherein, the resource allocation model is dynamically maintained based on the target resource allocation result corresponding to the beam at the historical moment.

[0008] In an implementation manner, determining the resource demand priority corresponding to each service to be transmitted at the current moment includes:

[0009] For the service to be transmitted uploaded by each user terminal, determining the delay factor corresponding to the service to be transmitted at the current moment, and taking the inner product between the data packet size corresponding to the service to be transmitted and the delay factor as the resource demand priority corresponding to the service to be transmitted;

[0010] Among them, there is a positive correlation between the resource demand priority and the data packet size and the delay factor.

[0011] In one implementation, beamforming is performed based on the resource demand priority corresponding to each service to be transmitted to determine the beam that can cover the target user terminal and the resource demand degree corresponding to the beam at the current moment, including:

[0012] Based on the resource demand priority corresponding to each service to be transmitted, multiple target user terminals are screened out from the user terminals;

[0013] Traverse each target user terminal, and divide the target user terminal and other target user terminals within the preset range of the target user terminal into the same user terminal cluster to obtain multiple user terminal clusters and the spatial shape and area value corresponding to each user terminal cluster;

[0014] For each user terminal cluster, beamforming is performed according to the spatial shape corresponding to the user terminal cluster to obtain a beam that can cover each target user terminal within the user terminal cluster, and based on the product of the maximum resource demand priority within the user terminal cluster and the area value corresponding to the user terminal cluster, the resource demand degree corresponding to the beam at the current moment is determined.

[0015] In one implementation, through the resource allocation model corresponding to the beam, according to the resource demand degree corresponding to the beam, the target resource allocation result corresponding to the beam at the current moment is determined, including:

[0016] For each beam, the resource demand degree corresponding to the beam is normalized to obtain the normalized resource demand degree corresponding to the beam at the current moment. Through the resource allocation model corresponding to the beam, based on the total number of beams at the current moment and the normalized resource demand degree corresponding to the beam, an initial resource allocation result and a reward value corresponding to the beam at the current moment are generated;

[0017] The initial resource allocation results corresponding to each beam at the current moment are normalized to obtain the target resource allocation results corresponding to each beam at the current moment.

[0018] In one implementation, after determining the target resource allocation result corresponding to the beam at the current moment through the resource allocation model corresponding to the beam according to the resource demand degree corresponding to the beam, the method further includes:

[0019] Taking the current moment as the new historical moment;

[0020] Based on the mapping relationship between the total number of beams and the experience pool, the experience data of each resource allocation model at the new historical moment is divided into the experience pool corresponding to the total number of beams at the new historical moment; wherein, the experience data includes the state space, action space, reward value of the resource allocation model, and the state space transferred after executing the action space, the state space includes the total number of beams and the normalized resource demand degree, and the action space includes the initial resource allocation result.

[0021] Based on the sampling probability corresponding to the experience data stored in the experience pool, target experience data is sampled from the experience pool, and each resource allocation model is dynamically maintained using the target experience data.

[0022] In one implementation, after sampling multiple target experience data from the experience pool based on the sampling probability corresponding to each experience data stored in the experience pool, the method further includes:

[0023] For any experience data stored in the experience pool, if the experience data is sampled as target experience data, the sampling probability corresponding to the experience data is lowered, and if the experience data is not sampled as target experience data, the sampling probability corresponding to the experience data is raised.

[0024] In one implementation, if the experience data is sampled as target experience data, the sampling probability corresponding to the experience data is lowered, and if the experience data is not sampled as target experience data, the sampling probability corresponding to the experience data is raised, including:

[0025] The sampling probability corresponding to the experience data is adjusted according to the following formula:

[0026] ;

[0027] Wherein, is the adjusted sampling probability corresponding to the th experience data in the th experience pool, is the sampling probability before adjustment corresponding to the th experience data in the th experience pool, indicates that the experience data is sampled as target experience data, indicates that the experience data is not sampled as target experience data, is the sampling loss, is the sampling quantity, is the experience pool size.

[0028] In a second aspect, the present invention further provides a phased array beamforming and hopping beam device for a QoE-driven low-earth orbit satellite, including:

[0029] A priority determination module, configured to receive services to be transmitted uploaded by multiple user terminals, and determine the resource requirement priority corresponding to each service to be transmitted at the current moment;

[0030] A beam shape design module, configured to perform beamforming based on the resource requirement priority corresponding to each service to be transmitted, so as to determine the beam that can cover the target user terminal and the resource requirement degree corresponding to the beam at the current moment;

[0031] A beam resource allocation module, configured to determine the target resource allocation result corresponding to the beam at the current moment through the resource allocation model corresponding to the beam according to the resource requirement degree corresponding to the beam, so as to transmit the service to be transmitted uploaded by the target user terminal according to the target resource allocation result; wherein, the resource allocation model is dynamically maintained based on the target resource allocation result corresponding to the beam at the historical moment.

[0032] Thirdly, the present invention further provides an electronic device, including a processor and a memory, where the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of the first aspect.

[0033] Fourthly, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by the processor, the computer-executable instructions cause the processor to implement the method according to any one of the first aspect.

[0034] A phase array beamforming and hopping beam method and device for a QoE-driven low-earth orbit satellite provided by the present invention first receives services to be transmitted uploaded by multiple user terminals, and determines the resource requirement priority corresponding to each service to be transmitted at the current moment; then performs beamforming based on the resource requirement priority corresponding to each service to be transmitted to determine the beam that can cover the target user terminal and the resource requirement degree corresponding to the beam at the current moment; finally, through the resource allocation model corresponding to the beam, according to the resource requirement degree corresponding to the beam, determines the target resource allocation result corresponding to the beam at the current moment, so as to transmit the service to be transmitted uploaded by the target user terminal according to the target resource allocation result; wherein, the resource allocation model is dynamically maintained based on the target resource allocation result corresponding to the beam at the historical moment. The above method determines the beam that can cover the target user terminal and its corresponding resource requirement degree through beamforming, and then combines the dynamically maintained resource allocation model to generate the corresponding target resource allocation result according to the resource requirement degree to realize the transmission of the service to be transmitted. The present invention can effectively save the resource waste caused by unnecessary coverage of areas without users, and significantly improve the downlink resource allocation performance of low-earth orbit satellites.

[0035] Other features and advantages of the present invention will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the present invention. The objectives and other advantages of the present invention are realized and attained by the structure particularly pointed out in the specification, claims as well as the drawings.

[0036] In order to make the above objectives, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, details are described as follows. Description of the Drawings

[0037] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0038] Figure 1 A flowchart of a phased array beamforming and beam hopping method for a QoE-driven low-earth orbit satellite provided by an embodiment of the present invention;

[0039] Figure 2 A technical framework diagram of a phased array beamforming and beam hopping method for a QoE-driven low-earth orbit satellite provided by an embodiment of the present invention;

[0040] Figure 3 A scenario diagram of a low-earth orbit satellite with a beam of variable shape provided by an embodiment of the present invention;

[0041] Figure 4 A comparison diagram of the downlink throughput of a satellite with the number of training times provided by an embodiment of the present invention;

[0042] Figure 5 A comparison diagram of the service queuing delay of the satellite downlink provided by an embodiment of the present invention;

[0043] Figure 6 A structural diagram of a phased array beamforming and beam hopping device for a QoE-driven low-earth orbit satellite provided by an embodiment of the present invention;

[0044] Figure 7 A structural diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] Currently, traditional beam hopping methods will cause power waste due to unnecessary coverage of areas without users, thereby affecting power utilization efficiency. Beamforming technology can generate beams with specific shapes and directivities by changing the phase and amplitude of the signals transmitted and received by antenna elements, and provide precise services for users to reduce the power loss caused by covering unnecessary areas, thereby improving the quality of experience (QoE) of users. Based on this, the embodiments of the present invention provide a QoE-driven phased array beamforming and beam hopping method and device for low-earth orbit satellites. This method combines phased array beamforming and beam hopping technologies, which can effectively save the resource waste caused by unnecessary coverage of areas without users. In addition, using a dynamically maintained resource allocation model to generate the target resource allocation results corresponding to each beam at each moment can significantly improve the downlink resource allocation performance of low-earth orbit satellites.

[0047] To facilitate the understanding of this embodiment, first, a QoE-driven phased array beamforming and beam hopping method for low-earth orbit satellites disclosed in the embodiments of the present invention will be introduced in detail. The operating environment of this method is a low-earth orbit satellite equipped with a uniformly planar beamformable digital-analog hybrid phased array, where the spacing between array elements is equal to half a wavelength and the array elements can freely adjust the phase of the transmitted signal. Refer to Figure 1 the flowchart of a QoE-driven phased array beamforming and beam hopping method for low-earth orbit satellites shown in the figure. This method mainly includes the following steps S102 to S106:

[0048] Step S102: Receive the services to be transmitted uploaded by multiple user terminals, and determine the resource demand priority corresponding to each service to be transmitted at the current moment.

[0049] In one example, after the user terminal accesses the low-earth orbit satellite, it will upload the service to be transmitted to the low-earth orbit satellite. The low-earth orbit satellite senses the frequency band resources, power resources, computing resources, etc. required for the service to be transmitted, and calculates the resource demand priority corresponding to each service to be transmitted at the current moment. The resource demand priority is positively correlated with the data packet size of the service to be transmitted and the delay factor of the service to be transmitted queued.

[0050] Step S104: Perform beamforming based on the resource requirement priority corresponding to each service to be transmitted, so as to determine the beam that can cover the target user terminal and the resource requirement degree corresponding to the beam at the current moment.

[0051] In one example, the target user terminals can be filtered based on the resource requirement priority corresponding to each service to be transmitted. The target user terminals are the user terminals that need to perform service transmission at the current moment. Cluster the target user terminals to obtain multiple user terminal clusters, and then plan the shape of the beam according to the spatial shape of the user terminal clusters, and determine the required resource degree corresponding to each beam at the current moment. The resource requirement degree is used to describe the power size and bandwidth size required by the beam. The higher the resource requirement degree, the greater the power and bandwidth required by the beam.

[0052] Step S106: Through the resource allocation model corresponding to the beam, according to the resource requirement degree corresponding to the beam, determine the target resource allocation result corresponding to the beam at the current moment, so as to transmit the service to be transmitted uploaded by the target user terminal according to the target resource allocation result.

[0053] Among them, the target resource allocation result is also the beam and bandwidth allocation scheme, which is used to describe the power and bandwidth that can be allocated to the beam at the current moment; the resource allocation model can adopt the multi-soft-actor-critic algorithm. Each beam corresponds to an actor-critic algorithm, so it is called the multi-soft-actor-critic algorithm. Each actor-critic algorithm needs to be dynamically maintained based on the target resource allocation result corresponding to the beam at the historical moment.

[0054] In one example, construct a state space according to the resource requirement degree corresponding to the beam. The state space includes the total number of beams at the current moment and the normalized resource requirement degree corresponding to the beam at the current moment; use the state space as the input of the actor-critic algorithm corresponding to the beam, so that the actor-critic algorithm outputs an action space, and the action space is also the initial resource allocation result corresponding to the beam; normalize the initial resource allocation results corresponding to all beams to obtain the target resource allocation result.

[0055] Furthermore, after generating the target resource allocation result, the current moment will be used as the new historical moment, and the experience data generated during the process of planning the resource allocation result will be incorporated into the corresponding experience pool (which can also be called the interaction experience group or interaction experience pool). The experience data is the aforementioned state space, action space, reward value, state space transferred after executing the action space, etc. The number of experience pools is multiple, and each experience pool is only used to store experience data consistent with the total number of beams corresponding to it; sample the experience data stored in the experience pool according to the sampling probability corresponding to each experience data, so as to dynamically maintain the actor-critic algorithm corresponding to each beam at the current moment.

[0056] Furthermore, the sampling probability can also be dynamically maintained. For example, if this experience data is sampled this time, the sampling probability corresponding to this experience data is decreased; if this experience data is not sampled this time, the sampling probability corresponding to this experience data is increased. When performing the next dynamic maintenance for the actor-critic algorithm, sampling can be performed from the experience data stored in the experience pool based on the adjusted sampling probability, so as to dynamically maintain the actor-critic algorithm corresponding to each beam at the current moment by using the sampled experience data.

[0057] The phased array beamforming and beam hopping method for low-earth orbit satellites driven by QoE provided by the embodiments of the present invention determines the beams that can cover the target user terminal and their corresponding resource requirements through beamforming, and then generates corresponding target resource allocation results according to the resource requirements in combination with the dynamically maintained resource allocation model to realize the transmission of the service to be transmitted. The embodiments of the present invention can effectively save the resource waste caused by unnecessary coverage of areas without users, and significantly improve the downlink resource allocation performance of low-earth orbit satellites.

[0058] For ease of understanding, the embodiments of the present invention first explain the technical framework of the phased array beamforming and beam hopping method for low-earth orbit satellites driven by QoE. Refer to Figure 2 the technical framework diagram of a phased array beamforming and beam hopping method for low-earth orbit satellites driven by QoE shown in the figure. It is divided into a satellite downlink transmission management module and a satellite downlink receiving user segment. The satellite downlink transmission management module further includes a beam resource scheduling decision general system, a beam frequency band management module, a beam power management module, an interactive experience pool management module, and a beamforming module. The satellite downlink receiving user segment further includes a mobile user device (i.e., the aforementioned user terminal) and a satellite signal receiving base station. Among them, the beam resource scheduling decision general system is used to implement the aforementioned steps S102 to S106, that is, scheme generation, and send the generated target resource allocation result to the beam frequency band management module and the beam power management module to realize the allocation of the beam frequency band and the beam power. On this basis, the beamforming module will interact with the mobile user device and the satellite signal receiving base station according to the target resource allocation result to realize the transmission of the service to be transmitted. At the same time, the beam resource scheduling decision general system is also used to collect interactive experience (i.e., the aforementioned experience data) and send the interactive experience to the interactive experience pool management module, and the interactive experience pool management module maintains the beam resource scheduling decision general system.

[0059] On the basis of Figure 2 this, the embodiments of the present invention provide a specific implementation manner of the phased array beamforming and beam hopping method for low-earth orbit satellites driven by QoE.

[0060] For the foregoing step S102, after receiving the services to be transmitted uploaded by multiple user terminals, the resource requirement priority corresponding to each service to be transmitted at the current moment can be determined according to the following process: for the services to be transmitted uploaded by each user terminal, determine the delay factor corresponding to the service to be transmitted at the current moment, and take the inner product of the packet size corresponding to the service to be transmitted and the delay factor as the resource requirement priority corresponding to the service to be transmitted.

[0061] Among them, the calculation process of the delay factor is as follows:

[0062] ;

[0063] Among them represents the length of a single time slot, represents the service queuing survival time. When the waiting delay of the queued service exceeds this service queuing survival time , the transmission task of this task will be determined to fail.

[0064] Among them, the calculation process of the resource requirement priority is as follows:

[0065] ;

[0066] Among them, represents the th user terminal, represents the th moment, represents the queue of packets to be transmitted of the th user terminal at the th moment (that is, the packet size of the service to be transmitted), represents the delay factor of the th user at the th moment.

[0067] In the embodiment of the present invention, by multiplying the delay factor of the service queuing to be transmitted and summing the packet sizes of the packets waiting to be transmitted in the service to be transmitted, it shows that the greater the delay of the service to be transmitted and the greater the packet size of the service to be transmitted, the greater the resource requirement priority of the service to be transmitted. After calculating the resource requirement priority, sort the services to be transmitted according to the resource requirement priority.

[0068] For the foregoing step S104, the embodiment of the present invention provides an implementation manner of beamforming based on the resource requirement priority corresponding to each service to be transmitted to determine the beam that can cover the target user terminal and the resource requirement degree corresponding to the beam at the current moment, including the following steps 1A to 1C:

[0069] Step 1A: Based on the resource requirement priorities corresponding to each service to be transmitted, multiple target user terminals are screened out from the user terminals.

[0070] In one example, after obtaining the sorted services to be transmitted, the first user terminals can be selected according to the needs of the low-earth orbit satellite load as target user terminals for clustering processing.

[0071] Step 1B: Traverse each target user terminal, and divide the target user terminal and other target user terminals within the preset range of the target user terminal into the same user terminal cluster, so as to obtain multiple user terminal clusters and the corresponding spatial shape and area value of each user terminal cluster.

[0072] In one example, the first user terminals are all marked as unclustered user terminals, and then each unclustered user terminal is traversed to find whether there are other user terminals nearby If there are, these user terminals are divided into clustered user terminals until all user terminals are traversed, and the corresponding spatial shape and the occupied area value of each user terminal cluster are obtained. .

[0073] Step 1C: For each user terminal cluster, beamforming is performed according to the corresponding spatial shape of the user terminal cluster to obtain a beam that can cover each target user terminal within the user terminal cluster, and based on the product of the maximum resource requirement priority within the user terminal cluster and the area value corresponding to the user terminal cluster, the resource requirement degree corresponding to the beam at the current moment is determined.

[0074] In one example, referring to Figure 3 a scenario diagram of a low-earth orbit satellite using a variable-shaped beam shown, Figure 3 it is shown that the user terminals on the ground are divided into multiple user terminal clusters, and the spatial shapes of each user terminal cluster are different. The low-earth orbit satellite can perform beamforming according to each spatial shape, that is, by adjusting the phase of the array elements in the digital-analog hybrid phased array equipped on the low-earth orbit satellite, the shape of the beam can be changed as needed to obtain a beam that can accurately cover each target user terminal within the corresponding user terminal cluster, thereby saving power waste caused by unnecessary coverage.

[0075] Furthermore, after determining each beam, the resource acquisition degree of each user terminal cluster can be calculated through the area value occupied by each user terminal cluster and the resource requirement priority of each target user terminal within the user terminal cluster. . Specifically, the calculation process of the resource acquisition degree is as follows:

[0076] ;

[0077] Among them, among them represents the area value of the th user terminal cluster at the th moment, represents the resource demand priority vector of each target user terminal within the user terminal cluster covered by the th beam. When the resource demand priority of the target user terminals included in the user terminal cluster is greater and the area value occupied by the user terminal cluster is larger, the power and bandwidth required by the beam are larger.

[0078] For the foregoing step S106, the embodiment of the present invention provides an implementation manner for determining the target resource allocation result corresponding to the beam at the current moment according to the resource demand degree corresponding to the beam through the resource allocation model corresponding to the beam, including the following steps 2A to step 2B:

[0079] Step 2A, for each beam, perform normalization processing on the resource demand degree corresponding to the beam to obtain the normalized resource demand degree corresponding to the beam at the current moment. Through the resource allocation model corresponding to the beam, based on the total number of beams at the current moment and the normalized resource demand degree corresponding to the beam, generate the initial resource allocation result and the reward value corresponding to the beam at the current moment.

[0080] Among them, the total number of beams at the current moment and the normalized resource demand degree corresponding to the beam can constitute the state space of the downlink data transmission of the low-earth orbit satellite. The state space is as follows:

[0081] ;

[0082] Among them, represents the state space of the th beam at the current moment , is the total number of beams at the current moment , is the normalized resource demand degree corresponding to the th beam at the current moment .

[0083] Take the above state space as the input of the actor-critic algorithm corresponding to the beam. The actor-critic algorithm will output the action space corresponding to the beam at the current moment, that is, the initial resource allocation result, and calculate the reward of the actor-critic algorithm. The action space of each satellite can be expressed as . Among them, is the beam power allocation scheme, It is the normalization of the beam bandwidth. The bandwidth and power allocation scheme with a large resource requirement degree of the beam will be preferentially executed. In order to synchronously optimize the throughput of the system, the queuing delay of the service to be transmitted, and the fairness of the system, the rewards are as follows:

[0084] ;

[0085] wherein, is the reward of the actor-critic algorithm corresponding to the th beam at the current moment , , , , represent the weights of each optimization objective, represents the number of user terminals covered by the th beam, is the throughput, is the maximum queuing delay vector, is the variance function, is the matrix composed of the maximum queuing delay vectors.

[0086] Through the design of the state space, action space, and reward value in the embodiments of the present invention, the actor-critic algorithm beam resource allocation module can be adopted to allocate the power and bandwidth allocation schemes for generating beams at each moment, and maintain the actor-critic algorithm after collecting a certain number of experiences. Through the actor-critic algorithm, the system periodically updates the frequency band allocation strategy to adapt to the changes in user requirements. The dynamic maintenance of the experience pool ensures the optimization of the frequency band resource allocation and the stable convergence of the algorithm.

[0087] Step 2B: Normalize the initial resource allocation result corresponding to each beam at the current moment to obtain the target resource allocation result corresponding to each beam at the current moment.

[0088] Among them, the normalized action space (i.e., the target resource allocation result) can be expressed as:

[0089] ;

[0090] wherein, is the target resource allocation result corresponding to the th beam at the current moment , is the normalization of the beam power, is the normalization of the beam bandwidth.

[0091] Furthermore, the embodiments of the present invention also provide an implementation manner for dynamically maintaining the actor-critic algorithm, including the following steps 3A to 3C:

[0092] Step 3A, taking the current moment as the new historical moment.

[0093] Step 3B, based on the mapping relationship between the total number of beams and the experience pool, dividing the experience data of each resource allocation model at the new historical moment into the experience pool corresponding to the total number of beams at the new historical moment.

[0094] Among them, the experience data includes the state space, action space, reward value of the resource allocation model, and the state space transferred after executing the action space. The state space includes the total number of beams and the normalized resource demand degree, and the action space includes the initial resource allocation result.

[0095] In one example, multiple experience pools are pre-configured, and the experience pools correspond one-to-one with the total number of beams. For example, when the total number of beams is 3, it corresponds to one experience pool, and when the total number of beams is 4, it corresponds to another experience pool. The experience pool can be expressed as:

[0096] ;

[0097] Among them, represents the number of beam clusters. Based on this, the target experience pool can be determined according to the total number of beams at the current moment, and the experience data of each actor-critic algorithm at the current moment can be divided into this target experience pool.

[0098] Step 3C, based on the sampling probability corresponding to the experience data saved in the experience pool, sampling the target experience data from the experience pool, and using the target experience data to dynamically maintain each resource allocation model.

[0099] In one example, if the total number of beams at the current moment is 3, sampling can be performed on the experience data saved in the experience pool corresponding to the total number of beams of 3 according to the dynamically adjusted sampling probability; if the total number of beams at a certain future moment is 4, sampling can be performed on the experience data saved in the experience pool corresponding to the total number of beams of 4 according to the dynamically adjusted sampling probability, and then using the sampled target experience data to train each actor-critic algorithm.

[0100] Furthermore, the embodiment of the present invention also provides an implementation method for dynamically adjusting the sampling probability, including: for any experience data saved in the experience pool, if the experience data is sampled as the target experience data, the sampling probability corresponding to the experience data is lowered, and if the experience data is not sampled as the target experience data, the sampling probability corresponding to the experience data is raised. Specifically, the sampling probability corresponding to the experience data can be adjusted according to the following formula:

[0101] ;

[0102] Among them, is the adjusted sampling probability corresponding to the th experience data in the th experience pool, is the sampling probability before adjustment corresponding to the th experience data in the th experience pool, indicates that the experience data is sampled as the target experience data, indicates that the experience data is not sampled as the target experience data, is the sampling loss, is the number of samples, is the size of the experience pool.

[0103] In practical applications, when a set of experience data is sampled, its probability will decrease , while the probabilities of other experience groups that are not sampled will increase , so as to ensure that the sampling probability of all experience data is 1. In addition, when new experience is added to the replay pool, the experience with the smallest sampling probability will be preferentially deleted.

[0104] Based on the foregoing embodiments, the embodiments of the present invention exemplarily provide a set of simulation parameters for low-earth orbit satellites, including: the orbital altitude is 500 Km; the antenna type is a planar phased array antenna, supporting beamforming and frequency hopping capabilities; the number of array elements is 16×16; the number of beams is 20; the frequency spectrum range is Ku; the bandwidth is 500 MHz; the bandwidth allocation per beam is 50 MHz - 200 MHz, dynamically adjusted based on beam requirements; the antenna gain is 20 dBi - 35 dBi; the total satellite power budget is 500 W.

[0105] In summary, the embodiments of the present invention have at least the following characteristics:

[0106] (1) Referring to Figure 4 a comparison graph of the throughput of a satellite downlink with the number of training times shown in Figure 5 and a comparison graph of the queuing delay of satellite downlink services shown in

[0107] Based on the foregoing embodiments, the embodiments of the present invention provide a QoE-driven phased array beamforming and hopping beam device for low-earth orbit satellites. Referring to Figure 6Schematic structural diagram of a phased array beamforming and hopping beam device for a QoE-driven low-earth orbit satellite. The device mainly includes the following parts:

[0108] A priority determination module 602, configured to receive services to be transmitted uploaded by multiple user terminals, and determine the resource demand priority corresponding to each service to be transmitted at the current moment;

[0109] A beam shape design module 604, configured to perform beamforming based on the resource demand priority corresponding to each service to be transmitted, so as to determine the beam that can cover the target user terminal and the resource demand degree corresponding to the beam at the current moment;

[0110] A beam resource allocation module 606, configured to determine the target resource allocation result corresponding to the beam at the current moment through the resource allocation model corresponding to the beam, according to the resource demand degree corresponding to the beam, so as to transmit the service to be transmitted uploaded by the target user terminal according to the target resource allocation result; wherein, the resource allocation model is dynamically maintained based on the target resource allocation result corresponding to the beam at the historical moment.

[0111] The phased array beamforming and hopping beam device for a QoE-driven low-earth orbit satellite provided by the embodiment of the present invention determines the beam that can cover the target user terminal and its corresponding resource demand degree through beamforming, and then combines the dynamically maintained resource allocation model to generate the corresponding target resource allocation result according to the resource demand degree, so as to realize the transmission of the service to be transmitted. The embodiment of the present invention can effectively save the resource waste caused by unnecessary coverage of areas without users, and significantly improve the downlink resource allocation performance of low-earth orbit satellites.

[0112] In one implementation, the priority determination module 602 is specifically configured to:

[0113] For each service to be transmitted uploaded by a user terminal, determine the delay factor corresponding to the service to be transmitted at the current moment, and use the inner product between the data packet size corresponding to the service to be transmitted and the delay factor as the resource demand priority corresponding to the service to be transmitted;

[0114] Wherein, the resource demand priority is positively correlated with the data packet size and the delay factor.

[0115] In one implementation, the beam shape design module 604 is specifically configured to:

[0116] Based on the resource demand priority corresponding to each service to be transmitted, screen out multiple target user terminals from the user terminals;

[0117] Traverse each target user terminal, and divide the target user terminal and other target user terminals within the preset range of the target user terminal into the same user terminal cluster, so as to obtain multiple user terminal clusters and the corresponding spatial shape and area value of each user terminal cluster;

[0118] For each user terminal cluster, perform beamforming according to the spatial shape corresponding to the user terminal cluster to obtain a beam that can cover each target user terminal within the user terminal cluster, and determine the resource demand degree corresponding to the beam at the current moment based on the product of the maximum resource demand priority within the user terminal cluster and the area value corresponding to the user terminal cluster.

[0119] In one implementation, the beam resource allocation module 606 is specifically used for:

[0120] For each beam, perform normalization processing on the resource demand degree corresponding to the beam to obtain the normalized resource demand degree corresponding to the beam at the current moment. Through the resource allocation model corresponding to the beam, based on the total number of beams at the current moment and the normalized resource demand degree corresponding to the beam, generate the initial resource allocation result and reward value corresponding to the beam at the current moment;

[0121] Perform normalization processing on the initial resource allocation result corresponding to each beam at the current moment to obtain the target resource allocation result corresponding to each beam at the current moment.

[0122] In one implementation, it further includes a model maintenance module, which is used for:

[0123] Take the current moment as the new historical moment;

[0124] Based on the mapping relationship between the total number of beams and the experience pool, divide the experience data of each resource allocation model at the new historical moment into the experience pool corresponding to the total number of beams at the new historical moment; where the experience data includes the state space, action space, reward value, and the state space transferred after executing the action space of the resource allocation model, the state space includes the total number of beams and the normalized resource demand degree, and the action space includes the initial resource allocation result;

[0125] Based on the sampling probability corresponding to the experience data stored in the experience pool, sample the target experience data from the experience pool, and use the target experience data to dynamically maintain each resource allocation model.

[0126] In one implementation, it further includes a sampling probability adjustment module, which is used for:

[0127] For any piece of experience data stored in the experience pool, if the experience data is sampled as the target experience data, the sampling probability corresponding to the experience data is decreased; if the experience data is not sampled as the target experience data, the sampling probability corresponding to the experience data is increased.

[0128] In one implementation, the sampling probability adjustment module is specifically configured to:

[0129] Adjust the sampling probability corresponding to the experience data according to the following formula:

[0130] ;

[0131] where is the adjusted sampling probability corresponding to the th piece of experience data in the th experience pool, is the sampling probability before adjustment corresponding to the th piece of experience data in the th experience pool, indicates that the experience data is sampled as the target experience data, indicates that the experience data is not sampled as the target experience data, is the sampling loss, is the number of samplings, is the size of the experience pool.

[0132] The device provided by the embodiments of the present invention has the same implementation principle and the same technical effects as those of the foregoing method embodiments. For the sake of brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding contents in the foregoing method embodiments.

[0133] The embodiments of the present invention provide an electronic device. Specifically, the electronic device includes a processor and a storage device; a computer program is stored on the storage device, and when the computer program is run by the processor, it executes the method according to any one of the foregoing implementations.

[0134] Figure 7 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes: a processor 70, a memory 71, a bus 72, and a communication interface 73. The processor 70, the communication interface 73, and the memory 71 are connected through the bus 72; the processor 70 is used to execute an executable module stored in the memory 71, such as a computer program.

[0135] Among them, the memory 71 may include a high-speed random access memory (RAM), or may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 73 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0136] The bus 72 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of easy representation, Figure 7 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] Among them, the memory 71 is used to store a program. After receiving an execution instruction, the processor 70 executes the program. The method executed by the device defined by the flow process disclosed in any embodiment of the foregoing embodiments of the present invention can be applied to the processor 70 or implemented by the processor 70.

[0138] The processor 70 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 70 or by instructions in software form. The above-mentioned processor 70 can be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it can also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 71, and the processor 70 reads the information in the memory 71 and combines its hardware to complete the steps of the above method.

[0139] The computer program product of the readable storage medium provided by the embodiments of the present invention includes a computer-readable storage medium storing program codes. The instructions included in the program codes can be used to execute the methods described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be elaborated here.

[0140] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

[0141] Finally, it should be noted that the above embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or easily conceive of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A QoE driven phased array beamforming and beam hopping method for a low-orbit satellite, characterized in that: include: Receiving services to be transmitted uploaded by multiple user terminals, and determining the resource demand priority corresponding to each of the services to be transmitted at the current moment; Performing beamforming based on the resource demand priority corresponding to each of the services to be transmitted to determine a beam that can cover a target user terminal and a resource demand corresponding to the beam at the current moment; Determine the target resource allocation result corresponding to the beam at the current moment according to the resource demand corresponding to the beam through the resource allocation model corresponding to the beam, so as to transmit the service to be transmitted uploaded by the target user terminal according to the target resource allocation result; wherein the resource allocation model is obtained by dynamically maintaining the target resource allocation result corresponding to the beam at a historical moment; Beamforming is performed based on the resource demand priority corresponding to each of the services to be transmitted to determine a beam that can cover a target user terminal and the resource demand corresponding to the beam at the current moment, including: based on the resource demand priority corresponding to each of the services to be transmitted, screening out multiple target user terminals from the user terminals; traversing each of the target user terminals, and dividing the target user terminal and other target user terminals located within a preset range of the target user terminal into the same user terminal cluster to obtain multiple user terminal clusters and a spatial shape and area value corresponding to each of the user terminal clusters; for each of the user terminal clusters, beamforming is performed according to the spatial shape corresponding to the user terminal cluster to obtain a beam that can cover each of the target user terminals in the user terminal cluster, and based on the product of the maximum resource demand priority in the user terminal cluster and the area value corresponding to the user terminal cluster, the resource demand corresponding to the beam at the current moment is determined.

2. The QoE-driven phased array beamforming and beam hopping method for low-orbit satellites according to claim 1, characterized in that: Determining the resource demand priority corresponding to each of the services to be transmitted at the current moment includes: For each of the services to be transmitted uploaded by the user terminal, determine a delay factor corresponding to the service to be transmitted at the current moment, and use an inner product between a data packet size corresponding to the service to be transmitted and the delay factor as a resource demand priority corresponding to the service to be transmitted; There is a positive correlation between the resource demand priority, the data packet size and the delay factor.

3. The QoE-driven phased array beamforming and beam hopping method for low-orbit satellites according to claim 1, characterized in that: Determining, by means of a resource allocation model corresponding to the beam and according to the resource demand corresponding to the beam, a target resource allocation result corresponding to the beam at the current moment, including: For each of the beams, normalize the resource demand corresponding to the beam to obtain a normalized resource demand corresponding to the beam at the current moment, and generate an initial resource allocation result and a reward value corresponding to the beam at the current moment based on the total number of beams at the current moment and the normalized resource demand corresponding to the beam through a resource allocation model corresponding to the beam; The initial resource allocation result corresponding to each of the beams at the current moment is normalized to obtain a target resource allocation result corresponding to each of the beams at the current moment.

4. The QoE-driven phased array beamforming and beam hopping method for low-orbit satellites according to claim 3, characterized in that: After determining the target resource allocation result corresponding to the beam at the current moment according to the resource demand corresponding to the beam through the resource allocation model corresponding to the beam, the method further includes: Taking said current moment as a new historical moment; Based on the mapping relationship between the total number of beams and the experience pool, the experience data of each resource allocation model at the new historical moment is divided into the experience pool corresponding to the total number of beams at the new historical moment; wherein the experience data includes the state space, action space, reward value and state space transferred after executing the action space of the resource allocation model, the state space includes the total number of beams and the normalized resource demand, and the action space includes the initial resource allocation result; Based on the sampling probability corresponding to the experience data stored in the experience pool, target experience data is sampled from the experience pool, and each resource allocation model is dynamically maintained using the target experience data.

5. The QoE-driven phased array beamforming and beam hopping method for low-orbit satellites according to claim 4, characterized in that: After sampling a plurality of target experience data from the experience pool based on the sampling probability corresponding to each experience data stored in the experience pool, the method further includes: For any of the experience data stored in the experience pool, if the experience data is sampled as the target experience data, the sampling probability corresponding to the experience data is lowered; if the experience data is not sampled as the target experience data, the sampling probability corresponding to the experience data is increased.

6. The QoE-driven phased array beamforming and beam hopping method for low-orbit satellites according to claim 5, characterized in that: If the experience data is sampled as the target experience data, then lowering the sampling probability corresponding to the experience data, and if the experience data is not sampled as the target experience data, then raising the sampling probability corresponding to the experience data, including: The sampling probability corresponding to the empirical data is adjusted according to the following formula: ; in, For the Experience pool The adjusted sampling probability corresponding to the empirical data is For the Experience pool The sampling probability before adjustment corresponding to the empirical data is Indicates that the experience data is sampled as the target experience data, Indicates that the experience data is not sampled as the target experience data, is the sampling loss, is the number of samples, is the experience pool size.

7. A QoE-driven phased array beamforming and beam hopping device for a low-orbit satellite, characterized in that: include: A priority determination module, used to receive services to be transmitted uploaded by multiple user terminals, and determine the resource demand priority corresponding to each service to be transmitted at the current moment; A beam shape design module, configured to perform beamforming based on the resource demand priority corresponding to each of the services to be transmitted, so as to determine a beam that can cover a target user terminal and a resource demand corresponding to the beam at the current moment; A beam resource allocation module, used to determine the target resource allocation result corresponding to the beam at the current moment according to the resource demand corresponding to the beam through the resource allocation model corresponding to the beam, so as to transmit the to-be-transmitted service uploaded by the target user terminal according to the target resource allocation result; wherein the resource allocation model is obtained by dynamically maintaining the target resource allocation result corresponding to the beam at a historical moment; The beam shape design module is specifically used to: screen out multiple target user terminals from the user terminals based on the resource demand priority corresponding to each of the services to be transmitted; traverse each of the target user terminals, and divide the target user terminal and other target user terminals within a preset range of the target user terminal into the same user terminal cluster to obtain multiple user terminal clusters and the spatial shape and area value corresponding to each of the user terminal clusters; for each of the user terminal clusters, perform beam shaping according to the spatial shape corresponding to the user terminal cluster to obtain a beam that can cover each of the target user terminals in the user terminal cluster, and determine the resource demand degree corresponding to the beam at the current moment based on the product of the maximum resource demand priority in the user terminal cluster and the area value corresponding to the user terminal cluster.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 6.

Images