Frequency resource allocation method and device based on reinforcement learning, equipment and medium

Abstract

The invention provides a frequency resource allocation method and device based on reinforcement learning, equipment and a medium, and relates to the technical field of wireless communication. The method comprises the following steps: acquiring environment state information of each beam cell covered by a target satellite at a first moment, the environment state information comprising user resource demand information and frequency resource occupation information; according to the environment state information of each beam cell at the first moment and the frequency selection information of each beam cell at the second moment, determining the environment state information of each beam cell at the second moment, and updating a corresponding reward value, the second moment being the next moment of the first moment; and circularly executing the steps until the reward value meets a preset condition, and obtaining a frequency resource allocation scheme of the target satellite. According to the invention, the frequency band distribution of the edge area users in the cell covered by the satellite can meet the dynamic flow request change, the signal-to-noise ratio of the edge users is improved, and the overall throughput of the system is improved.

Description

Background Art

[0002] In the case of conventional geostationary satellites, the satellite's beam coverage is often stationary relative to the ground, and the proportion of edge users remains basically unchanged. However, for satellites in rapid motion (for example, low-orbit satellites), the beam coverage of users is constantly changing, and the number of edge users and their resource requirements change frequently. At this time, the satellite often needs to use a more flexible resource allocation method to adapt to the changing needs of edge users.

[0003] In the related art, the partial frequency reuse method is mainly used, that is, the entire frequency band is divided into two parts: the central sub-band and the edge sub-band, wherein the central sub-band is used for central user scheduling, and the frequency reuse factor is 1, and the edge sub-band is used for edge user scheduling, and the frequency reuse factor is greater than 1. At this time, the spectrum allocation of users in the edge area is fixed at the beginning of system deployment, resulting in a lack of dynamic adjustment mechanism for the upper limit of the bandwidth available to edge users. With the high-speed movement of low-orbit satellites, the needs of edge users are constantly changing, and the static partial frequency reuse method cannot dynamically adapt to user needs.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a frequency resource allocation method, device, equipment and medium based on reinforcement learning, which at least to a certain extent overcomes the problem in the related art that satellites cannot dynamically allocate frequency bands and thus cannot meet the frequency resource requirements of edge users.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a frequency resource allocation method based on reinforcement learning is provided, including: obtaining environmental status information of each beam cell covered by a target satellite at a first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information; determining the environmental status information of each beam cell at the second moment according to the environmental status information of each beam cell at the first moment and the frequency selection information at a second moment, and updating the corresponding reward value, wherein the second moment is the next moment of the first moment; and looping through the above steps until the reward value meets a preset condition to obtain a frequency resource allocation scheme for the target satellite.

[0008] In some embodiments, before selecting the frequency selection information of each beam cell at the first moment and the frequency selection information at the second moment according to the environment status information of each beam cell at the first moment, the method further includes: obtaining the frequency selection information of each beam cell at the second moment based on a preset action selection strategy.

[0009] In some embodiments, the frequency selection information of each beam cell at the second moment is obtained by the following formula:

[0010]

[0011] Among them, a t represents the frequency band selection of the cell at time t, s t represents the environmental status information of the cell at time t, Q(s t ,a t ) represents the Q value at time t, argmax(·) represents the index value of the maximum value in (·), ε represents the probability, which is a constant less than 1, and random represents the random selection of actions.

[0012] In some embodiments, based on the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment, the environmental status information of each beam cell at the second moment is determined, and the corresponding reward value is updated, including: inputting the environmental status information of each beam cell at the first moment and the frequency selection information of the second moment into a pre-constructed decision model, outputting the environmental status information of each beam cell at the second moment, and updating the corresponding reward value.

[0013] In some embodiments, the above steps are executed in a loop until the reward value meets a preset condition, and a frequency resource allocation plan for the target satellite is obtained, including: executing the above steps in a loop at a preset time interval, and recording the environmental status information and frequency selection information of each beam cell at the current moment obtained after each loop, until the reward value meets the preset condition, and a frequency resource allocation plan for the target satellite is obtained.

[0014] In some embodiments, the reward value is expressed by the following formula:

[0015]

[0016] Among them, R t represents the reward function at time t, represents the edge user satisfaction of cell n at time t, Q n represents the number of resource blocks allocated to the private subband in cell n, C n,edge represents the edge user in cell n, u represents the subscript of the edge user in cell n, and d u It represents the number of resource blocks required by edge user u in cell n.

[0017] According to another aspect of the present disclosure, a frequency resource allocation device based on reinforcement learning is also provided, including: an information acquisition module, used to obtain the environmental status information of each beam cell covered by the target satellite at the first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information; a reward value updating module, used to determine the environmental status information of each beam cell at the second moment according to the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment, and update the corresponding reward value, wherein the second moment is the next moment of the first moment; a scheme generation module, used to repeatedly execute the above steps until the reward value meets the preset conditions to obtain the frequency resource allocation scheme of the target satellite.

[0018] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned reinforcement learning-based frequency resource allocation methods by executing the executable instructions.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the frequency resource allocation method based on reinforcement learning described above is implemented.

[0020] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein when the computer program is executed by a processor, the reinforcement learning-based frequency resource allocation method described above is implemented.

[0021] The frequency resource allocation method, device, equipment and medium based on reinforcement learning provided in the embodiments of the present disclosure, the embodiments of the present disclosure determine the environmental state information at the second moment through the environmental state information of each beam cell covered by the target satellite at the first moment and the frequency selection information at the second moment, and update the corresponding reward value, and repeat the above steps until the reward value meets the preset conditions to obtain the frequency resource allocation scheme of the target satellite. The embodiments of the present disclosure can make the frequency band allocation of users in the edge area of ​​the cell covered by the satellite meet the dynamic traffic request changes, improve the signal-to-noise ratio of edge users, and improve the overall throughput of the system.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1 A schematic diagram of an exemplary application system architecture to which the frequency resource allocation method based on reinforcement learning in an embodiment of the present disclosure can be applied is shown;

[0025] Figure 2 A flow chart of a frequency resource allocation method based on reinforcement learning in an embodiment of the present disclosure is shown;

[0026] Figure 3 A flow chart of another frequency resource allocation method based on reinforcement learning in an embodiment of the present disclosure is shown;

[0027] Figure 4 A schematic diagram of a frequency resource allocation device based on reinforcement learning in an embodiment of the present disclosure is shown;

[0028] Figure 5 A block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0032] Figure 1FIG. 1 shows an exemplary application system architecture diagram to which the frequency resource allocation method based on reinforcement learning in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include cell a 1011 , cell b 1012 , cell c 1013 and target satellite 102 .

[0033] Among them, the cell users in each cell can be divided into central users and edge users according to their location. For example, they can be divided according to the distance D between the cell users and the beam center. Users with a distance greater than D belong to the edge user set C. n,edge , that is, the blank part in the figure, users whose distance is less than or equal to D belong to the central user set C n,central , i.e. the shaded part in the figure, where n is the subscript of the cell.

[0034] It should be noted that the distance D between the cell user and the beam center can generally be regarded as 2 / 3 of the cell radius, and the embodiment of the present disclosure does not specifically limit the value of the distance D between the cell user and the beam center. Similarly, the embodiment of the present disclosure can adopt any method that can distinguish between the central user and the edge user, and the embodiment of the present disclosure does not specifically limit the method for distinguishing between the central user and the edge user.

[0035] The multiple beams emitted by satellite 102 cover cells a1011, b1012 and c1013 respectively. The disclosed embodiment defines states (environmental state information of each cell at different times), actions (frequency band selection information of each cell at different times) and reward values ​​(satisfaction of users at the edge of each cell), establishes an optimization problem, and models the optimization problem as a Markov decision model to meet the dynamic demand of users at the edge of each cell for frequency resources.

[0036] Those skilled in the art will know that Figure 1 The number of cells and satellites in the example is only illustrative, and any number of cells and satellites may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0037] Under the above system architecture, an embodiment of the present disclosure provides a frequency resource allocation method based on reinforcement learning, which can be executed by any electronic device with computing and processing capabilities.

[0038] In some embodiments, the reinforcement learning-based frequency resource allocation method provided in the embodiments of the present disclosure may be executed by a base station device in a cell of the above-mentioned system architecture; in other embodiments, the reinforcement learning-based frequency resource allocation method provided in the embodiments of the present disclosure may be executed by a base station device on a satellite in the above-mentioned system architecture; in other embodiments, the reinforcement learning-based frequency resource allocation method provided in the embodiments of the present disclosure may be implemented by a base station device in a cell and a base station device on a satellite in the above-mentioned system architecture through interaction.

[0039] Figure 2 A flow chart of a frequency resource allocation method based on reinforcement learning in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the method comprises the following steps:

[0040] S202, obtaining environmental status information of each beam cell covered by the target satellite at the first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information.

[0041] In one embodiment of the present disclosure, each beam provided by the target satellite will cover multiple beam cells. First, it is necessary to obtain the environmental status information of each beam cell at the first moment, wherein the environmental status information may include user resource demand information and frequency resource occupancy information. The user resource demand information can generally be expressed as the number of resource blocks required in the edge area of ​​each beam cell at the first moment, and the frequency resource occupancy information can generally be expressed as the occupancy status of each resource block at the first moment.

[0042] S204, determining the environmental status information of each beam cell at the second moment according to the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment, and updating the corresponding reward value, wherein the second moment is the next moment of the first moment.

[0043] In one embodiment of the present disclosure, when a certain beam cell selects a frequency resource in a frequency range, it will affect other beam cells. Therefore, the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment (i.e., the next moment of the first moment) are used to determine the environmental status information of each beam cell at the next moment, and the corresponding reward value is updated at the same time. The reward value can be a custom value. For example, the partial frequency reuse scheme desired in the embodiment of the present disclosure can meet the dynamic resource requirements of users in the edge area of ​​the beam cell, that is, the reward value can be set to the satisfaction of users in the edge area.

[0044] It should be noted that the reward value may be any data that can be used to reflect the dynamic resource demand of users in the edge area of ​​the beam cell, and the embodiment of the present disclosure does not specifically limit the content of the reward value.

[0045] S206, the above steps are executed repeatedly until the reward value meets the preset condition, and a frequency resource allocation scheme for the target satellite is obtained.

[0046] In one embodiment of the present disclosure, the above steps need to be cycled to optimize the result. A standard value of the reward value can be preset as a sign of stopping the cycle, or the number of cycles can be preset, and the cycle is stopped when the preset number of cycles is reached. The number of cycles here is often determined based on actual experience or the effect of data update, and the present embodiment does not specifically limit this.

[0047] As can be seen from the above, the disclosed embodiment determines the environmental status information at the second moment through the environmental status information at the first moment and the frequency selection information at the second moment of each beam cell covered by the target satellite, and updates the corresponding reward value, and repeats the above steps until the reward value meets the preset conditions to obtain the frequency resource allocation plan of the target satellite. The disclosed embodiment can make the frequency band allocation of users in the edge area of ​​the cell covered by the satellite meet the dynamic traffic request changes, improve the signal-to-noise ratio of edge users, and improve the overall throughput of the system.

[0048] In one embodiment of the present disclosure, data is first initialized and defined. Assuming that the target satellite is a low-orbit satellite, the low-orbit satellite provides N beams to cover N beam cells in the area. The set of beam cells can be expressed as C set ={c n |n=1,2,...,N},where c n represents the nth cell; the total bandwidth can be expressed as B tot The minimum unit for resource allocation in the system is a resource block RB, and its bandwidth can be expressed as B min The total number of resource blocks RB can be expressed as K = B tot / B min , the number of the resource block set can be expressed as R set ={k|k=1, 2, ..., K}, where k represents the kth resource block RB.

[0049] In one embodiment of the present disclosure, the cell users can be divided into central users and edge users according to their locations in the beam cell. There are often multiple division methods, for example, division based on the distance D between the cell users and the beam center. When the distance between the cell users and the beam center is greater than D, the cell users belong to the edge user set C. n，edge ; When the distance between the cell user and the beam center is less than or equal to D, the cell user belongs to the central user set C n，central , where n is the subscript of the beam cell.

[0050] In one embodiment of the present disclosure, the distance D between the cell user and the beam center can generally be set to 2 / 3 of the cell radius, and can also be adjusted according to actual conditions. The embodiment of the present disclosure does not specifically limit the actual value of the distance D between the cell user and the beam center.

[0051] In an embodiment of the present disclosure, central users and marginal users may be divided in any manner according to actual conditions. On the premise that the division manner is reasonable, the embodiment of the present disclosure does not specifically limit the manner of dividing user types.

[0052] In one embodiment of the present disclosure, a private subband may generally refer to a portion of the spectrum used exclusively by a specific user or entity, and a shared subband may generally refer to a portion of the spectrum that can be shared by multiple users or entities. The frequency band allocation scheme of cell n can be expressed as F n =[f n，1 , f n，2 , …, f n，k , …, f n，K ], where f n，k represents the allocation of the kth resource block relative to cell n. n，k =0, it means that the kth resource block has not been allocated; if f n，k =1, it means that the kth resource block is allocated to cell n and used as a private subband; if f n，k =2, it means that the kth resource block is allocated to cell n and used as a shared subband.

[0053] In one embodiment of the present disclosure, the satellite part frequency reuse scheme is expressed by the following formula:

[0054]

[0055] Among them, S represents the frequency reuse scheme of low-orbit satellites, F N represents the partial frequency reuse scheme implemented by the low-orbit satellite for the Nth cell, f N，K Indicates the allocation of the Kth resource block relative to the Nth cell.

[0056] In an embodiment of the present disclosure, before the above S204, the method further includes: acquiring frequency selection information of each beam cell at the second moment based on a preset action selection strategy, wherein the preset action selection strategy is a greedy strategy.

[0057] In one embodiment of the present disclosure, an action may be defined in, Indicates the frequency band selection of the nth cell at time t.

[0058] In one embodiment of the present disclosure, the frequency selection information of each beam cell at the second moment is obtained by the following formula:

[0059]

[0060] Among them, a t represents the frequency band selection of the cell at time t, s t represents the environmental status information of the cell at time t, Q(s t , a t ) represents the Q value at time t, argmax(·) represents the index value of the maximum value in (·), ε represents the probability, which is a constant less than 1, and random represents the random selection of actions.

[0061] In one embodiment of the present disclosure, the greedy strategy is generally an algorithm strategy that takes the best, optimal or most favorable choice in the current state in each step of selection, thereby leading to the best result. The greedy strategy is often simple and efficient, and can obtain a local optimal solution. In order to avoid the situation where the greedy strategy may never select a certain action during the decision-making process, the above formula (2) is used to make the band selection process keep exploring new actions, that is, in most cases (that is, when the probability is 1-ε), the band selection is performed according to the calculated Q value table, and in individual cases (that is, when the probability is ε), the action is randomly selected to improve the reliability of the result.

[0062] In one embodiment of the present disclosure, the above S204 includes: inputting the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment into a pre-constructed decision model, outputting the environmental status information of each beam cell at the second moment, and updating the corresponding reward value, wherein the pre-constructed decision model is a Markov decision model.

[0063] In one embodiment of the present disclosure, the main goal of the Markov decision process is to find a strategy that can maximize the expected reward over a long period of time. In order to make the frequency band allocation of edge area users in a cell covered by a satellite meet the dynamic traffic request changes, the present embodiment can establish an optimization problem with the goal of maximizing the satisfaction of edge users of the system, define states, actions and rewards, and model the optimization problem as a Markov decision model.

[0064] In one embodiment of the present disclosure, the state may be represented by the environmental state information of the cell. Specifically, the environmental state information of the cell n may be represented as in, The matrix represents the number of resource blocks required by the edge area of ​​each cell at time t, Indicates the number of resource blocks required in the edge area of ​​the nth cell, represents the resource occupancy matrix of the resource block at time t, represents the resource occupancy of the kth resource block at time t, where It means that the kth resource block is idle at time t, and cell n can occupy the resource block at this time; if This means that the kth resource block at time t has been occupied by other cells.

[0065] In one embodiment of the present disclosure, the above S206 includes: executing the above steps in a loop at a preset time interval, and recording the environmental status information and frequency selection information of each beam cell at the current moment obtained after each loop, until the reward value meets the preset conditions, and the frequency resource allocation plan of the target satellite is obtained.

[0066] In one embodiment of the present disclosure, the reward value is expressed by the following formula:

[0067]

[0068] Among them, R t represents the reward function at time t, represents the edge user satisfaction of cell n at time t, Q n represents the number of resource blocks allocated to the private subband in cell n, C n，edge represents the edge user in cell n, u represents the subscript of the edge user in cell n, and d u It represents the number of resource blocks required by edge user u in cell n.

[0069] In one embodiment of the present disclosure, the optimization problem can be expressed by the following formula:

[0070]

[0071] in, represents the edge user satisfaction of cell n at time t, N represents the total number of cells, n represents the cell subscript, K n，com represents the resource block number set allocated to the center area of ​​cell n, K n，pri Indicates the resource block number set allocated to the edge area of ​​cell n, K m，pri Indicates the resource block number set allocated to the edge area of ​​cell m.

[0072] In one embodiment of the present disclosure, the optimization direction of the optimization problem is limited by three preset constraints. Under the premise that cell m and cell n are adjacent, the first constraint condition indicates that the resource block set allocated to cell n does not exceed the total resource block set of the system, the second constraint condition indicates that the resource block sets allocated to the edge area and the central area of ​​cell n are isolated from each other, and the third constraint condition indicates that the resource block sets allocated to the edge areas of cell n and cell m are isolated from each other.

[0073] In one embodiment of the present disclosure, the Q value update rule can be expressed by the following formula:

[0074]

[0075] Among them, Q(s t , a t ) represents the Q value at time t, Q(s t+1 , a t+1 ) represents the Q value at time t+1, α represents the learning rate, R t+1 represents the reward value at time t+1, γ represents the discount rate, max(·) is used to calculate the maximum value, s t represents the environmental status information of the cell at time t, s t+1 represents the environmental status information of the cell at time t+1, a t represents the frequency band selection of the cell at time t, a t+1 Indicates the frequency band selection of the cell at time t+1.

[0076] In one embodiment of the present disclosure, the satellite provides services to users, and the cell selects the frequency band according to the current environmental status information, obtains the updated reward value, and updates the Q value according to the above formula (5). At the same time, the environmental status information of the cell changes. According to the changed environmental status information of the cell, the above steps are repeated again, and the Q value updated in each time slot is recorded to form a Q value table until the reward value meets the preset conditions. At this time, the Q value table converges stably, and the satellite partial frequency division multiplexing scheme at all times can be obtained. Reinforcement learning pursues long-term returns, finds the best decision through the interaction between the intelligent agent and the environment, and adapts to the dynamic changes of the environment in real time. The embodiment of the present disclosure proposes a satellite dynamic partial frequency multiplexing method based on reinforcement learning through reinforcement learning, so that the frequency band allocation of users in the edge area meets the dynamic changes in traffic requests, improves the signal-to-noise ratio of edge users, and improves the overall throughput of the system.

[0077] It should be noted that the preset number of iterations can also be set as the stopping condition of the loop. The appropriate number of training times helps to improve the accuracy of the results, but there is no definite value. The preset number of iterations can be set according to actual conditions or past historical experience, and the embodiments of the present disclosure do not make specific limitations on this.

[0078] Figure 3 A flow chart of another frequency resource allocation method based on reinforcement learning in an embodiment of the present disclosure is shown. Figure 3 As shown, the method comprises the following steps:

[0079] S301, initializing the data and Q value table when the satellite provides services to the users in the cell.

[0080] In one embodiment of the present disclosure, the satellite provides N beams to cover N beam cells in the area, and the set of beam cells can be expressed as C set ={c n |n=1,2,...,N},where c n represents the nth cell; the total bandwidth can be expressed as B tot The minimum unit for resource allocation in the system is a resource block RB, and its bandwidth can be expressed as B min The total number of resource blocks RB can be expressed as K = B tot / B min , the number of the resource block set can be expressed as R set ={k|k=1, 2, ..., K}, where k represents the kth resource block RB.

[0081] In one embodiment of the present disclosure, the cell users can be divided into central users and edge users according to their locations in the beam cell. There are often multiple division methods, for example, division based on the distance D between the cell users and the beam center. When the distance between the cell users and the beam center is greater than D, the cell users belong to the edge user set C. n，edge ; When the distance between the cell user and the beam center is less than or equal to D, the cell user belongs to the central user set C n，central , where n is the subscript of the beam cell.

[0082] In one embodiment of the present disclosure, the Q value table is constructed based on the Q value updated in each time slot.

[0083] S302, obtaining the initial environmental state information of the cell and the first moment, taking the environmental state of the cell as the state of the agent in the reinforcement learning process, and defining the reward value in the reinforcement learning process.

[0084] In one embodiment of the present disclosure, the environmental status information of cell n can be expressed as Specifically, The matrix represents the number of resource blocks required by the edge area of ​​each cell at time t, Indicates the number of resource blocks required in the edge area of ​​the nth cell, represents the resource occupancy matrix of the resource block at time t, represents the resource occupancy of the kth resource block at time t, where It means that the kth resource block is idle at time t, and cell n can occupy the resource block at this time; if This means that the kth resource block at time t has been occupied by other cells.

[0085] In one embodiment of the present disclosure, the reward value can be expressed by the above formula (3). The optimization problem in the reinforcement learning process is set according to the reward value, and the optimization problem can be expressed by the above formula (4).

[0086] S303, obtaining the frequency band selection information of the cell at a second moment, and using the frequency band selection of the cell as the action of the agent in the reinforcement learning process, wherein the second moment is the next moment of the first moment.

[0087] In one embodiment of the present disclosure, an action may be defined in, Indicates the frequency band selection of the nth cell at time t.

[0088] S304, establishing a Markov decision model, determining the environmental status information of each cell at the second moment according to the environmental status information of each cell at the first moment and the frequency selection information at the second moment, and updating the Q value and the reward value.

[0089] In one embodiment of the present disclosure, the Q value may be updated by the above formula (5).

[0090] S305, obtaining frequency selection information at the next moment according to the greedy strategy, determining the environmental status information of each cell at the next moment, and updating the Q value and reward value again.

[0091] S306, determine whether the reward value reaches a preset threshold. If yes, execute S307; if no, execute S305.

[0092] S307, generating a partial frequency reuse scheme of the satellite for each cell at each time.

[0093] In one embodiment of the present disclosure, partial frequency reuse is a method of reducing co-channel interference and increasing the overall capacity of the system and the quality of user experience by allocating different frequency sets in certain areas.

[0094] As can be seen from the above, the disclosed embodiment determines the environmental status information at the second moment through the environmental status information at the first moment and the frequency selection information at the second moment of each beam cell covered by the target satellite, and updates the corresponding reward value, and repeats the above steps until the reward value meets the preset conditions to obtain the frequency resource allocation plan of the target satellite. The disclosed embodiment can make the frequency band allocation of users in the edge area of ​​the cell covered by the satellite meet the dynamic traffic request changes, improve the signal-to-noise ratio of edge users, and improve the overall throughput of the system.

[0095] Based on the same inventive concept, the embodiment of the present disclosure also provides a frequency resource allocation device based on reinforcement learning, as described in the following embodiment. Since the principle of solving the problem in the embodiment of the device is similar to that in the above method embodiment, the implementation of the embodiment of the device can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0096] Figure 4 A schematic diagram of a frequency resource allocation device based on reinforcement learning in an embodiment of the present disclosure is shown. Figure 4 As shown, the device includes: an information acquisition module 401, a reward value updating module 402 and a solution generating module 403.

[0097] Among them, the information acquisition module 401 is used to obtain the environmental status information of each beam cell covered by the target satellite at the first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information; the reward value updating module 402 is used to determine the environmental status information of each beam cell at the second moment according to the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment, and update the corresponding reward value, wherein the second moment is the next moment of the first moment; the scheme generation module 403 is used to repeatedly execute the above steps until the reward value meets the preset conditions to obtain the frequency resource allocation scheme of the target satellite.

[0098] As can be seen from the above, the disclosed embodiment determines the environmental status information at the second moment through the environmental status information at the first moment and the frequency selection information at the second moment of each beam cell covered by the target satellite, and updates the corresponding reward value, and repeats the above steps until the reward value meets the preset conditions to obtain the frequency resource allocation plan of the target satellite. The disclosed embodiment can make the frequency band allocation of users in the edge area of ​​the cell covered by the satellite meet the dynamic traffic request changes, improve the signal-to-noise ratio of edge users, and improve the overall throughput of the system.

[0099] In one embodiment of the present disclosure, data is first initialized and defined. Assuming that the target satellite is a low-orbit satellite, the low-orbit satellite provides N beams to cover N beam cells in the area. The set of beam cells can be expressed as C set ={cn |n=1,2,...,N},where c n represents the nth cell; the total bandwidth can be expressed as B tot The minimum unit for resource allocation in the system is a resource block RB, and its bandwidth can be expressed as B min The total number of resource blocks RB can be expressed as K = B tot / B min , the number of the resource block set can be expressed as R set ={k|k=1, 2, ..., K}, where k represents the kth resource block RB.

[0100] In one embodiment of the present disclosure, the cell users can be divided into central users and edge users according to their locations in the beam cell. There are often multiple division methods, for example, division based on the distance D between the cell users and the beam center. When the distance between the cell users and the beam center is greater than D, the cell users belong to the edge user set C. n，edge ; When the distance between the cell user and the beam center is less than or equal to D, the cell user belongs to the central user set C n，central , where n is the subscript of the beam cell.

[0101] In one embodiment of the present disclosure, the distance D between the cell user and the beam center can generally be set to 2 / 3 of the cell radius, and can also be adjusted according to actual conditions. The embodiment of the present disclosure does not specifically limit the actual value of the distance D between the cell user and the beam center.

[0102] In an embodiment of the present disclosure, central users and marginal users may be divided in any manner according to actual conditions. On the premise that the division method is reasonable and correct, the embodiment of the present disclosure does not specifically limit the method of dividing user types.

[0103] In one embodiment of the present disclosure, a private subband may generally refer to a portion of the spectrum used exclusively by a specific user or entity, and a shared subband may generally refer to a portion of the spectrum that can be shared by multiple users or entities. The frequency band allocation scheme of cell n can be expressed as F n =[f n，1 , f n2 , …, f n，k , …, f n，K ], where f n，k represents the allocation of the kth resource block relative to cell n. n，k =0, it means that the kth resource block has not been allocated; if f n，k =1, it means that the kth resource block is allocated to cell n and used as a private subband; if f n，k=2, it means that the kth resource block is allocated to cell n and used as a shared subband. The satellite part frequency reuse scheme can be expressed by the above formula (1).

[0104] In one embodiment of the present disclosure, the device further includes: an action selection module 404, configured to obtain frequency selection information of each beam cell at the second moment based on a preset action selection strategy, wherein the preset action selection strategy is a greedy strategy. The frequency selection information of each beam cell at the second moment can be obtained by the above formula (2).

[0105] In one embodiment of the present disclosure, the greedy strategy is generally an algorithm strategy that takes the best, optimal or most favorable choice in the current state in each step of selection, thereby leading to the best result. The greedy strategy is often simple and efficient, and can obtain a local optimal solution. In order to avoid the situation where the greedy strategy may never select a certain action during the decision-making process, the above formula (2) is used to make the band selection process keep exploring new actions, that is, in most cases (when the probability is 1-ε), the band selection is performed according to the calculated Q value table, and in individual cases (when the probability is ε), the action is randomly selected to improve the reliability of the result.

[0106] In one embodiment of the present disclosure, the above-mentioned reward value updating module 402 is also used to input the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment into a pre-constructed decision model, output the environmental status information of each beam cell at the second moment, and update the corresponding reward value, wherein the pre-constructed decision model is a Markov decision model.

[0107] In one embodiment of the present disclosure, the state may be represented by the environmental state information of the cell. Specifically, the environmental state information of the cell n may be represented as Specifically, The matrix represents the number of resource blocks required by the edge area of ​​each cell at time t, Indicates the number of resource blocks required in the edge area of ​​the nth cell, represents the resource occupancy matrix of the resource block at time t, represents the resource occupancy of the kth resource block at time t, where It means that the kth resource block is idle at time t, and cell n can occupy the resource block at this time; if This means that the kth resource block at time t has been occupied by other cells.

[0108] In one embodiment of the present disclosure, the solution generation module 403 is further used to cyclically execute the above steps at a preset time interval, and record the environmental status information and frequency selection information of each beam cell at the current moment obtained after each cycle, until the reward value meets the preset conditions, and the frequency resource allocation solution of the target satellite is obtained. The reward value can be expressed by the above formula (3).

[0109] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0110] Refer to the following Figure 5 The electronic device 500 according to this embodiment of the present disclosure is described. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0111] Figure 5 A block diagram of an electronic device in an embodiment of the present disclosure is shown. Figure 5 The electronic device 500 according to this embodiment of the present disclosure is described. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0112] like Figure 5 As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510).

[0113] The storage unit stores a program code, and the program code can be executed by the processing unit 510, so that the processing unit 510 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 510 can execute the following steps of the above method embodiment: obtain the environmental status information of each beam cell covered by the target satellite at the first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information; determine the environmental status information of each beam cell at the second moment according to the environmental status information of each beam cell at the first moment and the frequency selection information at the second moment, and update the corresponding reward value, wherein the second moment is the next moment of the first moment; repeat the above steps until the reward value meets the preset conditions, and obtain the frequency resource allocation plan of the target satellite.

[0114] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0115] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5205, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0116] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0117] The electronic device 500 may also communicate with one or more external devices 540 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 560. As shown, the network adapter 560 communicates with other modules of the electronic device 500 via a bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0119] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned frequency resource allocation methods based on reinforcement learning is implemented. Since the implementation of the computer-readable storage medium embodiment can refer to the implementation of the above-mentioned method embodiment, the repeated parts will not be repeated.

[0120] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0122] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0123] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0124] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0125] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0126] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A frequency resource allocation method based on reinforcement learning, characterized in that: include: Obtaining environmental status information of each beam cell covered by the target satellite at a first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information; Determine the environmental state information of each beam cell at the second moment according to the environmental state information of each beam cell at the first moment and the frequency selection information at the second moment, and update the corresponding reward value, wherein the second moment is a moment next to the first moment; The above steps are executed repeatedly until the reward value meets the preset condition, and the frequency resource allocation scheme of the target satellite is obtained.

2. The frequency resource allocation method based on reinforcement learning according to claim 1, characterized in that: Before selecting the frequency information at the second moment according to the environment status information of each beam cell at the first moment, the method further includes: Based on a preset action selection strategy, frequency selection information of each beam cell at the second moment is obtained.

3. The frequency resource allocation method based on reinforcement learning according to claim 2, characterized in that: The frequency selection information of each beam cell at the second moment is obtained by the following formula: Among them, a t represents the frequency band selection of the cell at time t, s t represents the environmental status information of the cell at time t, Q(s t ,a t ) represents the Q value at time t, argmax(·) represents the index value of the maximum value in (·), ε represents the probability, which is a constant less than 1, and random represents the random selection of actions.

4. The frequency resource allocation method based on reinforcement learning according to claim 1, characterized in that: Determining the environmental state information of each beam cell at the second moment according to the environmental state information of each beam cell at the first moment and the frequency selection information at the second moment, and updating the corresponding reward value, including: The environmental status information of each beam cell at the first moment and the frequency selection information at the second moment are input into a pre-built decision model, the environmental status information of each beam cell at the second moment is output, and the corresponding reward value is updated.

5. The frequency resource allocation method based on reinforcement learning according to claim 1, characterized in that: The above steps are executed cyclically until the reward value meets the preset condition, and the frequency resource allocation scheme of the target satellite is obtained, including: The above steps are executed repeatedly at preset time intervals, and the environmental status information and frequency selection information of each beam cell at the current moment obtained after each cycle are recorded until the reward value meets the preset conditions, and the frequency resource allocation plan of the target satellite is obtained.

6. The frequency resource allocation method based on reinforcement learning according to claim 5, characterized in that: The reward value is expressed by the following formula: Among them, R t represents the reward function at time t, represents the edge user satisfaction of cell n at time t, Q n represents the number of resource blocks allocated to the private subband in cell n, C n,edge represents the edge user in cell n, u represents the subscript of the edge user in cell n, and d u It represents the number of resource blocks required by edge user u in cell n.

7. A frequency resource allocation device based on reinforcement learning, characterized in that: include: An information acquisition module, used to acquire environmental status information of each beam cell covered by the target satellite at a first moment, wherein the environmental status information includes: user resource demand information and frequency resource occupancy information; A reward value updating module, configured to determine the environment state information of each beam cell at the second moment according to the environment state information of each beam cell at the first moment and the frequency selection information at the second moment, and update the corresponding reward value, wherein the second moment is a moment next to the first moment; The scheme generating module is used to execute the above steps repeatedly until the reward value meets the preset conditions, thereby obtaining the frequency resource allocation scheme of the target satellite.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the frequency resource allocation method based on reinforcement learning as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the frequency resource allocation method based on reinforcement learning described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, the frequency resource allocation method based on reinforcement learning as described in any one of claims 1 to 6 is implemented.

Images