Ultra-dense networking multi-service slice resource allocation method and device

Abstract

The present invention provides a method and device for allocating resources for multiple service slices in an ultra-dense networking, comprising: obtaining a multi-agent reinforcement learning model, which deploys a policy network and a value network on each micro base station, pre-solves a transmission power balancing solution, the policy network uses the micro base station transmission rate and transmission power as state parameters, and uses a set of associated parameters of each micro base station and a predicted set of transmission powers of other micro base stations as action parameters; each micro base station obtains its own state parameters and generates a corresponding action strategy, and the value network calculates an estimated Q value for the action strategy generated for the corresponding micro base station based on global information, which is used to update the policy network parameters; constructs a loss function of the estimated Q value and the actual Q value with the goal of maximizing the reward value, and updates the parameters of the value network until the model reaches the preset performance requirements; inputs the state parameters of each micro base station into the trained multi-agent reinforcement learning model, generates a corresponding action strategy, and realizes multi-service slice resource allocation.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to a method and device for allocating resources for multi-service slices in ultra-dense networking. Background Art

[0002] Network slicing technology has become one of the key technologies of the fifth generation mobile network (5G). In the next generation of mobile networks and other technical fields, the network has put forward higher requirements for flexibility, isolation, privacy and customization for users' differentiated services. At the same time, the importance of small-scale networks that provide specific services will also increase to meet the needs of different scenarios and different groups of people.

[0003] One of the emerging solutions is to introduce heterogeneous ultra-dense networking of macro and micro base stations to meet users' transmission capacity and coverage requirements. Ultra-dense networking of base stations can improve the spectrum efficiency of the system to a certain extent, and dynamically allocate wireless resources through fast resource scheduling, and reuse the licensed spectrum of macro base stations (MBS) at micro base stations (SBS), thereby improving the utilization of system wireless resources and spectrum efficiency. However, it also brings about system interference and system cost issues. In order to provide reliable services, micro base stations need to obtain the reuse license of macro base station spectrum, which requires interference coordination between macro base stations and micro base stations to ensure that their operations are not affected by harmful interference.

[0004] Therefore, there is an urgent need for a method to reduce micro base station competition interference and optimize wireless resource allocation while ensuring user communication quality and communication needs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for allocating resources for multi-service slices in an ultra-dense networking to eliminate or improve one or more defects existing in the prior art, and to solve the problems of system interference and system cost caused by the prior art when improving the utilization rate of system wireless resources and spectrum efficiency.

[0006] On the one hand, the present invention provides a method for allocating multi-service slice resources in an ultra-dense network, characterized in that the ultra-dense network includes at least one macro base station, each macro base station is also connected to serve multiple micro base stations; users of the micro base station reuse the slice resources of the corresponding macro base station, and the method is based on the cross-layer interference between the micro base station and the macro base station, and the same-layer interference between adjacent micro base stations. Perform multi-service slice resource allocation; the method comprises the following steps:

[0007] A multi-agent reinforcement learning model is obtained, wherein the multi-agent reinforcement learning model deploys a policy network and a value network on each micro base station; each policy network constructs a state space with the transmission rate and total transmission power of each user in the corresponding single micro base station as state parameters; an associated parameter indicating whether the user in each micro base station reuses the resource block in the macro base station is obtained, and an action space is constructed with the associated parameter set of each micro base station and the predicted transmission power set of each other micro base station as action parameters; each micro base station obtains its own state parameters, selects a corresponding action according to the policy network, and the value network of each micro base station generates an estimated Q value according to the state parameters and selected action of the corresponding micro base station and the state parameters and actions of other micro base stations, and is used to update the parameters of the policy network of the corresponding micro base station; a loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization reward value as the optimization target, and the parameters of the value network are updated; until the preset performance requirements are met;

[0008] Among them, in the state update process, the macro base station constructs a macro base station profit calculation formula according to the cross-layer interference price and the cross-layer interference generated by the user multiplexing the resource block in the micro base station; the micro base station constructs a micro base station profit calculation formula according to the associated parameters, the fixed bandwidth length of the resource block, the signal to interference plus noise ratio, the same-layer interference price, the same-layer interference, the cross-layer interference price and the cross-layer interference; the macro base station is taken as the leader and each micro base station is taken as the follower to construct a non-cooperative game; the value of the associated parameter is fixed, and the reverse induction method is used to solve the micro base station profit calculation formula to obtain the transmission power equilibrium solution of each micro base station, which is used to update the state space of each strategy network; the transmission power equilibrium solution is substituted into the macro base station profit calculation formula to obtain the cross-layer interference price equilibrium solution;

[0009] The state parameters of each micro base station are input into the multi-agent reinforcement learning model to generate corresponding action strategies to achieve multi-service slice resource allocation.

[0010] In some embodiments of the present invention, the macro base station constructs a macro base station revenue calculation formula according to the cross-layer interference price and the cross-layer interference generated by the user multiplexing resource blocks in the micro base station, and the macro base station revenue calculation formula is:

[0011]

[0012] Among them, U MBS represents the macro base station revenue; U UE represents the user set of all micro base stations; U PRB Indicates the total number of resource blocks; U BS represents the set of the macro base station and all micro base stations; represents the cross-layer interference price of user i using resource block j in micro base station b; It represents the cross-layer interference caused by user i using resource block j in micro base station b.

[0013] In some embodiments of the present invention, the micro base station constructs a micro base station revenue calculation formula based on the association parameters, the fixed bandwidth length of the resource block, the signal to interference plus noise ratio, the same-layer interference price, the same-layer interference, the cross-layer interference price and the cross-layer interference, and the micro base station revenue calculation formula is:

[0014]

[0015] st

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] Among them, U b represents the micro base station revenue; U UE,b represents the user set of micro base station b; U s Indicates the type of slice; represents the association between user i and slice s, resource block j, and micro base station b; B represents the fixed bandwidth length of the resource block; represents the signal to interference plus noise ratio caused by user i using resource block j in micro base station b; represents the same-layer interference price of user i using resource block j at micro base station b; represents the intra-layer interference caused by user i using resource block j in micro base station b; represents the cross-layer interference price of user i using resource block j in micro base station b; represents the cross-layer interference caused by user i using resource block j in micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; U UE represents the user set of all micro base stations; U BS represents the set of the macro base station and all micro base stations; I max Indicates the maximum interference value; U PRB represents the total number of resource blocks; τ represents the total number of resource blocks.

[0023] In some embodiments of the present invention, each strategy network constructs a state space using the transmission rate and total transmit power of each user in a single micro base station as state parameters, wherein the total transmit power uses the transmit power equalization solution, and the state parameters are expressed as:

[0024]

[0025] Among them, s j (t) represents the state parameters of the micro base station at time t; r N,j (t) represents the transmission rate of the Nth user multiplexing resource block j at time t; The transmission power of user i allocated to resource block j at micro base station b*; U j represents the set of users that multiplex resource block j;

[0026] The value network of each micro base station generates an estimated Q value according to the state parameters and selected actions of the corresponding micro base station and the state parameters and actions of other micro base stations. The state parameters of the value network are expressed as:

[0027] s j '(t)=(s j (t),a j (t),s -j (t),a -j (t));

[0028] Among them, s j (t) represents the state parameter of the micro base station at time t; a j (t) represents the action parameters of the micro base station at time t; s -j (t) represents the state parameter set of other micro base stations at time t; a -j (t) represents the action parameter set of other micro base stations at time t.

[0029] In some embodiments of the present invention, an association parameter for indicating whether a user in each micro base station reuses a resource block in a macro base station is obtained, and an action space is constructed using an association parameter set of each micro base station and a predicted transmit power set of each other micro base station as an action parameter, and the action parameter is expressed as:

[0030] a j (t) = {W j ,P -j};

[0031] in,

[0032]

[0033]

[0034] Among them, aj (t) represents the action parameters of the micro base station at time t; W j Represents a collection of associated parameters; P -j represents a set of predicted transmit powers of other micro base stations; represents the association relationship between user i and slice s, resource block j, and micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; U j represents the set of users that multiplex resource block j.

[0035] In some embodiments of the present invention, the value network of each micro base station generates an estimated Q value based on the state parameters and selected actions of the corresponding micro base station and the state parameters and actions of other micro base stations, and constructs a policy gradient based on the estimated Q value, and is used to update the parameters of the policy network of the corresponding micro base station. The calculation formula of the policy gradient is:

[0036]

[0037] in, represents the policy gradient; θ represents the policy parameter; J(u j ) represents the cumulative estimated reward value; D represents the experience replay pool; u j (a j |s j ) represents the action strategy made by the micro base station according to the status; Represents the value network; j Indicates the state of the micro base station estimated by the value network; a j represents the action of the micro base station estimated by the value network; other Indicates the status of other micro base stations estimated by the value network; a other Represents the actions of other micro base stations predicted by the value network.

[0038] In some embodiments of the present invention, a loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization of the reward value as the optimization goal, and the parameters of the value network are updated. The calculation formula of the loss function is:

[0039]

[0040] in, represents the loss function; θ represents the strategy parameter; u j represents the adaptive weight parameter; r represents the updated result of the action; Represents the value network; j Indicates the state of the micro base station estimated by the value network; a j represents the action of the micro base station estimated by the value network;other Indicates the status of other micro base stations estimated by the value network; a other Indicates actions of other micro base stations estimated by the value network; represents the actual Q value.

[0041] In some embodiments of the present invention, the loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization of the reward value as the optimization goal, and the parameters of the value network are updated. The calculation formula of the reward value is:

[0042]

[0043] Among them, reward j represents the reward value; u j represents the adaptive weight parameter; r j represents the total transmission rate of each user multiplexing resource block j of the micro base station; N represents the total number of users; r -j Indicates the total transmission rate of other micro base stations; represents the same-layer interference price; Indicates same-layer interference; represents the cross-layer interference price; represents cross-layer interference; U j represents the set of users that multiplex resource block j.

[0044] In some embodiments of the present invention, the adaptive weight parameter is learned based on the state of the global environment;

[0045] When u j =1, the reward value is only related to the transmission rate of the micro base station itself, which is a zero-sum game;

[0046] When 0 j <1, the reward value is not only related to the transmission rate of the micro base station itself, but also to the transmission rates of other micro base stations, forming a mixed game.

[0047] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods mentioned above.

[0048] The beneficial effects of the present invention are at least:

[0049] The present invention provides a method and device for allocating multi-service slice resources in an ultra-dense networking. By obtaining a pre-trained multi-agent reinforcement learning model, the state parameters of each micro base station are input into the multi-agent reinforcement learning model, and an action strategy is generated accordingly. This reduces the competition and interference of micro base stations while ensuring the communication quality and communication needs of users, thereby optimizing wireless resource allocation and alleviating spectrum tension.​

[0050] In the training of the multi-agent reinforcement learning model, the macro base station revenue calculation formula and the micro base station revenue calculation formula are pre-constructed, and the resource allocation problem is modeled as a non-cooperative game to obtain the transmission power equilibrium solution and the cross-layer interference price equilibrium solution; the transmission power equilibrium solution is then used to update the state space of each policy network in the multi-agent reinforcement learning model, guiding the model to update and optimize in the preset direction, simplifying the model's computational complexity. At the same time, the multi-agent reinforcement learning model deploys a policy network and a value network on each micro base station. The value network can obtain global information and generate a more accurate estimated Q value, allowing the policy network to generate a better action strategy.

[0051] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0052] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0054] Figure 1 This is a structural diagram of a method for allocating resources for multi-service slices in ultra-dense networking in one embodiment of the present invention.

[0055] Figure 2 Schematic diagram of ultra-dense networking architecture in one embodiment of the present invention.

[0056] Figure 3 This is a flow chart of a method for allocating resources for multi-service slices in ultra-dense networking in one embodiment of the present invention.

[0057] FIG4( a ) is a schematic diagram of base station revenue when the number of resource blocks is 20 in one embodiment of the present invention.

[0058] FIG4( b ) is a schematic diagram of base station revenue when the number of resource blocks is 24 in one embodiment of the present invention.

[0059] FIG4( c ) is a schematic diagram of base station revenue when the number of resource blocks is 28 in one embodiment of the present invention.

[0060] Figure 5It is a schematic diagram comparing the effects of the MADDPG algorithm and the Stackelberg game on the impact of cross-layer interference price on the average revenue of base stations in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0062] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0063] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0064] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0065] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0066] In order to solve the problems of system interference and system cost caused by the prior art when improving the system wireless resource utilization and spectrum efficiency, that is, to solve the problem of dynamic multi-service slicing and power allocation when macro base stations and micro base stations share the macro base station network in ultra-dense networking, the present invention provides an ultra-dense networking multi-service slicing resource allocation method, the ultra-dense networking includes at least one macro base station, and each macro base station is also connected to serve multiple micro base stations. The method includes the following steps S101-S102:

[0067] Step S101: Obtain a multi-agent reinforcement learning model, which is trained using a preset method.

[0068] Step S102: Input the state parameters of each micro base station into the multi-agent reinforcement learning model to generate corresponding action strategies to achieve multi-service slice resource allocation.

[0069] like Figure 1As shown, the present invention sets prices and charges by studying the cross-layer interference caused by macro base stations in the reuse of resource blocks of micro base stations. After receiving the pricing plan of the macro base station, each micro base station performs dynamic slicing and power allocation. Without affecting the macro base station users, the economic benefits and communication quality of the macro base station users and the micro base station users are improved. The problem is decomposed into a transmission power allocation problem and a slice resource block allocation problem, and the problem is converted into an optimization problem of maximizing the benefits of macro base stations and micro base stations under the premise of ensuring user communication quality and reducing interference.

[0070] Figure 2 This is a schematic diagram of an ultra-dense networking architecture. The ultra-dense networking network includes a macro base station and multiple micro base stations. The macro base station covers the range of multiple micro base stations. Users of the micro base stations can reuse the slice resources of the macro base station for communication, but this will cause cross-layer interference to the macro base station users. The slice manager is responsible for coordinating the number of resource blocks required for each slice to achieve isolated communication between slices. Slices can be deployed in multiple micro base stations. In order to reduce communication signaling overhead, each micro base station independently and distributedly allocates resources to its own micro base station and connected users. It should be noted that Figure 1 and Figure 2 Only one set of ultra-dense networking architecture is shown as a reference. In actual use, the number of macro base stations and micro base stations can be adjusted according to specific needs and / or specific application scenarios.

[0071] In step S101, in order to obtain a multi-agent reinforcement learning model suitable for the application scenario of the present invention, the initial reinforcement learning model needs to be trained. However, considering that it is difficult to directly train the initial reinforcement learning model to learn the resource allocation strategy, the model may not be optimized in the preset direction, and thus a better strategy cannot be obtained, and the training amount and the amount of calculation are too large. Therefore, in order to better describe the user's resource allocation strategy and simplify the amount of calculation of the initial reinforcement learning model, the present invention first uses the reverse induction method to analyze the problem, solve the game equilibrium, guide the initial reinforcement learning model to be trained and optimized in the preset direction, and finally obtain the multi-agent reinforcement learning model required by the present invention to achieve resource allocation.

[0072] The method for training the initial reinforcement learning model to obtain a multi-agent reinforcement learning model includes the following steps:

[0073] A strategy network and a value network are deployed on each micro base station; each strategy network constructs a state space with the transmission rate and total transmission power of each user in the corresponding single micro base station as state parameters; obtains associated parameters for indicating whether the user in each micro base station reuses the resource block in the macro base station, and constructs an action space with the associated parameter set of each micro base station and the predicted transmission power set of other micro base stations as action parameters; each micro base station obtains its own state parameters, selects the corresponding action according to the strategy network, and the value network of each micro base station generates an estimated Q value according to the state parameters and selected action of the corresponding micro base station and the state parameters and actions of other micro base stations, and is used to update the parameters of the strategy network of the corresponding micro base station; a loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization reward value as the optimization goal, and the parameters of the value network are updated until the preset performance requirements are met.

[0074] Among them, during the state update process, the macro base station constructs a macro base station profit calculation formula according to the cross-layer interference price and the cross-layer interference generated by the user multiplexing the resource block in the micro base station; the micro base station constructs a micro base station profit calculation formula according to the associated parameters, the fixed bandwidth length of the resource block, the signal to interference plus noise ratio, the same-layer interference price, the same-layer interference, the cross-layer interference price and the cross-layer interference; the macro base station is taken as the leader and each micro base station is taken as the follower to construct a non-cooperative game; the value of the associated parameters is fixed, and the reverse induction method is used to solve the micro base station profit calculation formula to obtain the transmission power equilibrium solution of each micro base station, which is used to update the state space of each strategy network; the transmission power equilibrium solution is substituted into the macro base station profit calculation formula to obtain the cross-layer interference price equilibrium solution.

[0075] Specifically, the revenue of the macro base station and the revenue of the micro base station form a set of Stackelberg games, in which the macro base station is the leader, responsible for setting the cross-layer interference price, and the micro base station is the follower, responsible for providing the associated parameters and transmission power. For the multi-objective joint optimization problem of this model, it is more difficult to obtain the game equilibrium solution and design the algorithm than the single-objective optimization problem, and the traditional iterative algorithm is not enough to solve the communication problem between multiple base stations. Therefore, in the present invention, the reverse induction method is adopted. First, the value of the associated parameter is fixed, and the calculation formula of the micro base station revenue is solved to obtain the transmission power equilibrium solution of each micro base station, and the result of the transmission power equilibrium solution is substituted into the macro base station revenue calculation formula to obtain the cross-layer interference price equilibrium solution. After obtaining the fixed equilibrium solution of the transmission power, the multi-agent reinforcement learning model is combined to provide an optimization scheme for resource block association, and the information of other micro base stations required for the equilibrium solution is provided through an independent prediction method. The overall process of optimizing the resource allocation strategy is shown in Figure (3).

[0076] In some embodiments, the cross-layer interference price set by the macro base station is The macro base station revenue calculation formula is shown in formula (1):

[0077]

[0078] In formula (1), U MBS U represents the revenue of macro base station; UE represents the user set of all micro base stations; U PRB Indicates the total number of resource blocks; U BS Represents the set of macro base stations and all micro base stations; represents the cross-layer interference price of user i using resource block j in micro base station b; It represents the cross-layer interference caused by user i using resource block j in micro base station b.

[0079] Among them, cross-layer interference The calculation formula is shown in formula (2):

[0080]

[0081] In formula (2), represents cross-layer interference; U s Indicates the type of slice; represents the association relationship between user i and slice s, slice resource block j, and micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; It represents the channel gain of user i in micro base station b multiplexing macro base station resource block j.

[0082] in, represents the association between user i and slice s, slice resource block j, and micro base station b. Specifically, it can be understood as follows: if user i uses resource block j belonging to slice s in micro base station b, then If user i does not use resource block j belonging to slice s in micro base station b,

[0083] From formula (1), we can see that the revenue of the macro base station is the sum of the charges for interference caused to the macro base station by the users of all micro base stations for transmission by multiplexing resource blocks. Each micro base station user has its own revenue, but multiplexing spectrum transmission will cause interference, which will affect the revenue of other micro base station users and macro base station users. Therefore, the macro base station is regarded as the leader and each micro base station as the follower, and the resource allocation problem of the micro base station is constructed as a non-cooperative game. Each micro base station user is considered to be a selfish and rational player. The strategy space of the game consists of the transmission power allocation strategy space and the slice resource block allocation strategy space. Each micro base station tries to maximize its utility.

[0084] In some embodiments, the calculation formula of the micro base station revenue is shown in formula (3), and its constraints are shown in formulas (4) to (9):

[0085]

[0086] st

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] In formula (3) to formula (9), U b represents the revenue of micro base station; U UE,b represents the user set of micro base station b; U s Indicates the type of slice; represents the association between user i and slice s, resource block j, and micro base station b; B represents the fixed bandwidth length of the resource block; represents the signal to interference plus noise ratio caused by user i using resource block j in micro base station b; represents the same-layer interference price of user i using resource block j at micro base station b; represents the intra-layer interference caused by user i using resource block j in micro base station b; represents the cross-layer interference price of user i using resource block j in micro base station b; represents the cross-layer interference caused by user i using resource block j in micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; U UE represents the user set of all micro base stations; U BS represents the set of macro base stations and all micro base stations; I max Indicates the maximum interference value; U PRB represents the total number of resource blocks; τ represents the total number of resource blocks.

[0094] Among the constraints, formula (4) indicates the lower limit of the transmission power of micro base station users; formula (5) indicates the maximum value of the same-layer interference of all users; formula (6) indicates the maximum value of the cross-layer interference of all users; formula (7) is the limit on the number of slice resource blocks, and the number of resource blocks allocated in all slices cannot exceed the total number of slice resource blocks; formula (8) indicates that the same user is associated with at most one base station; formula (9) is the isolation constraint of the slice, indicating that a resource block can only be allocated to the same user at the same time.

[0095] In formula (3), the same-layer interference The calculation formula is shown in formula (10):

[0096]

[0097] In formula (10), K b',b A binary variable indicating whether micro base station b' overlaps or is adjacent to micro base station b; represents the association relationship between user i and slice s, resource block j, and micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; represents the channel gain of resource block j of user i in micro base station b'.

[0098] The method for solving the transmit power balance solution and the cross-layer interference price balance solution includes the following steps:

[0099] Fixed association parameters To maximize the micro base station revenue, the micro base station revenue calculation formula, that is, formula (3), is simplified to:

[0100]

[0101] Constraint 1:

[0102] Constraint 2:

[0103] To U b Taking the derivative, we get:

[0104]

[0105] Taking the second-order derivative, we get:

[0106]

[0107] Therefore, U b It is a convex function on the domain, where σ represents the channel white noise.

[0108] Depend on have to:

[0109]

[0110] In this step, the transmit power balance solution of each micro base station is obtained.

[0111] Depend on The constraints can be obtained:

[0112]

[0113] make Let's assume

[0114] It is also necessary to ensure that the macro base station benefits are maximized. The calculation formula for the macro base station benefits is expressed as:

[0115]

[0116] Constraint 1:

[0117] Constraint 2:

[0118] Macro Base Station Revenue and Cross-Layer Interference Price It is related to the value interval of , which is a non-continuous optimization problem. Therefore, the indicator function is first introduced:

[0119]

[0120] Given When U MBS becomes a completely differentiable function. In the Stackelberg game, it is assumed that the macro base station is capable of collecting channel information between the macro base station and each micro base station, that is, (The channel gain of user i in micro base station b multiplexing macro base station resource block j), then the calculation formula of macro base station benefit can be rewritten as follows:

[0121]

[0122] Constraint 1:

[0123] Constraint 2:

[0124] Constraint 3:

[0125] And the above has been obtained

[0126] make

[0127] Then the KKT condition can be written as:

[0128]

[0129]

[0130]

[0131]

[0132] α,β,γ≥0;

[0133] From the KKT condition, we can get:

[0134]

[0135] Analysis shows that α=β=0.

[0136] Among them, the KKT condition is a form of the universal Lagrange multiplier method, which is mainly used as an optimal solution when there are unequal value constraints on the optimization function.

[0137] In summary, the equilibrium solution of the cross-layer interference price of the macro base station can be obtained as follows:

[0138] (1) When hour,

[0139]

[0140] (2) When hour,

[0141]

[0142] .......

[0143] (N) When hour,

[0144]

[0145]

[0146]

[0147] Thus, a set of Stackelberg game solutions are obtained: and Because in the given and hour, and The solution is unique, so it is a Nash equilibrium solution.

[0148] After obtaining the fixed equilibrium solution of the transmission power, the state space of the multi-agent reinforcement learning model strategy network is set, and the multi-agent reinforcement learning model is used to provide resource block association parameters. The information of other base stations required for the equilibrium solution is provided through independent prediction methods, and resource allocation optimization is performed in combination with the game results.

[0149] In some embodiments, the multi-agent reinforcement learning model uses the MADDPG algorithm, which is an extension of DDPG in multi-agent tasks. Its basic idea is centralized learning and decentralized execution. When training the model, the MADDPG algorithm introduces a critic that can observe global information to guide actor training, and only uses actors with local observations to take actions during testing.

[0150] The multi-agent reinforcement learning model deploys a strategy network (strategy generated by actors) and a value network (strategy generated by actors evaluated by critics) on each micro base station. Actors can only obtain information about their own micro base stations, while critics can obtain information about all micro base stations. Since each micro base station is in a non-cooperative competitive relationship in the present invention, the goals of each micro base station are different, and each micro base station has a corresponding strategy network and value network.

[0151] In some embodiments, each strategy network constructs a state space using the transmission rate and total transmission power of each user in a single micro base station as state parameters. The state parameter design is shown in formula (11):

[0152]

[0153] In formula (11), r N,j (t) represents the transmission rate of the Nth user multiplexing resource block j at time t; The transmission power of user i allocated to resource block j at micro base station b*; U j represents the set of users that multiplex resource block j.

[0154] First N items r 1,j (t),r 2,j (t),…,r N,j (t) represents the transmission rate of all users multiplexing resource block j at time t; the N+1th item represents the sum of the transmission powers allocated to all users multiplexing resource block j.

[0155] in, The transmit power balance solution obtained above is:

[0156]

[0157] In some embodiments, an association parameter indicating whether a user in each micro base station reuses a resource block in a macro base station is obtained, and an action space is constructed using an association parameter set of each micro base station and a predicted transmit power set of other micro base stations as action parameters. The action parameter design is shown in formula (12):

[0158] a j (t) = {Wj ,P -j}; (12)

[0159] In formula (11), W j express A collection of associated parameters; P -j represents the set of predicted transmit powers of other micro base stations, as shown in formula (13) and formula (14):

[0160]

[0161]

[0162] in, represents the association relationship between user i and slice s, resource block j, and micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; U j represents the set of users that multiplex resource block j.

[0163] In some embodiments, the value network of each micro base station generates an estimated Q value based on the state parameters and selected actions of the corresponding micro base station and the state parameters and actions of other micro base stations. The state parameters of the value network are shown in formula (15):

[0164] s j '(t)=(s j (t),a j (t),s -j (t),a -j (t)); (15)

[0165] In formula (15), s j (t) represents the state parameter of the micro base station at time t; a j (t) represents the action parameters of the micro base station at time t; s -j (t) represents the state parameter set of other micro base stations at time t; a -j (t) represents the action parameter set of other micro base stations at time t.

[0166] During the training of the multi-agent reinforcement learning model, each micro base station (actor) randomly samples according to the current state, selects and executes the corresponding action. Correspondingly, the value network (critic) calculates the estimated Q value according to the state of the micro base station and the selected action as feedback for the action of the micro base station. The policy network updates the policy according to the feedback of the critic, and the value network constructs a loss function based on the estimated Q value and the actual Q value for training. In the present invention, the critic can obtain global information, that is, it can obtain the state and action of other micro base stations to obtain a more accurate estimated Q value.

[0167] During the testing of the multi-agent reinforcement learning model, each micro base station performs random sampling based on the current state, selects and executes the corresponding action. At this time, there is no need for critic feedback, and there is no need to rely on the state or action of other micro base stations, thus achieving decentralized execution.

[0168] In the present invention, a state parameter is inputted, and a certain action is obtained, so it is a deterministic strategy.

[0169] For the deterministic strategy, the policy gradient used to update the policy network is constructed based on the estimated Q value of the value network. The policy gradient calculation formula is shown in formula (16):

[0170]

[0171] In formula (16), θ represents the policy parameter; J(u j ) represents the cumulative estimated reward value; D represents the experience replay pool; u j (a j |s j ) represents the action strategy made by the micro base station according to the status; represents the value network; s j represents the state of the micro base station estimated by the value network; a j represents the action of the micro base station estimated by the value network; s other represents the status of other micro base stations estimated by the value network; a other Represents the actions of other micro base stations estimated by the value network.

[0172] For the value network, it is necessary to construct a loss function between the estimated Q value and the actual Q value of the model with maximizing the reward value as the optimization goal to update the parameters.

[0173] For setting the reward value, firstly, the total transmission rate of the multiplexed resource blocks of each user of the micro base station is calculated, and the calculation formula is shown in formula (17):

[0174]

[0175] In formula (17), U j represents the set of users that reuse resource block j; r i,j represents the transmission rate of user i using resource block j; U s Indicates the type of slice; represents the association between user i and slice s, slice resource block j, and micro base station b; B represents the fixed bandwidth length of the resource block; It represents the signal to interference plus noise ratio caused by user i using resource block j in micro base station b.

[0176] in, is the transmit power equalization solution obtained above.

[0177] represents the transmission power of user i allocated to slice resource block j at micro base station b*; represents the channel gain of user i in resource block j of micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b'; represents the channel gain of resource block j of user i in micro base station b'.

[0178] On this basis, in order to express the competition and cooperation relationship between the micro base station and other micro base stations, the reward value calculation formula is designed as shown in formula (18):

[0179]

[0180] In formula (18), u j represents the adaptive weight parameter; r j represents the total transmission rate of each user multiplexing resource block j of the micro base station; N represents the total number of users; r -j Indicates the transmission rate of other micro base stations; represents the same-layer interference price; Indicates same-layer interference; represents the cross-layer interference price; represents cross-layer interference; U j represents the set of users that multiplex resource block j.

[0181] Specifically, the adaptive weight parameters are learned based on the state of the global environment. j =1, the reward value is only related to the transmission rate of the micro base station itself, which is a zero-sum game; when 0 j When <1, the reward value is not only related to the transmission rate of the micro base station itself, but also to the transmission rates of other micro base stations, forming a mixed game.

[0182] The loss function constructed based on the estimated Q value and the actual Q value of the model is shown in formula (19):

[0183]

[0184] In formula (19), θ represents the strategy parameter; u j represents the adaptive weight parameter; r represents the updated result of the action; represents the value network; s j represents the state of the micro base station estimated by the value network; a j represents the action of the micro base station estimated by the value network; s other represents the status of other micro base stations estimated by the value network; a​other represents the actions of other micro-base stations estimated by the value network; Indicates the actual Q value.

[0185] The updated calculation formula is shown in formula (20):

[0186]

[0187] In formula (20), r j represents the total transmission rate of resource block j of each user of the micro base station; represents the actual value network; s' j Indicates the actual state of the micro base station; a' j Indicates the actual action of the micro base station; s' other Indicates the actual status of other micro base stations; a' other Indicates the actual actions of other micro base stations; u' j Represents the actual adaptive weight parameter; o j Represents the local information observed by the micro base station.

[0188] Then the parameters are updated with reward values, and the parameters of the value network are updated with the maximization of reward value as the optimization goal; the policy network constructs a policy gradient according to the estimated Q value generated by the value network, and is updated according to the policy gradient to make the generated action strategy more accurate, until the initial reinforcement learning model reaches the preset performance requirements, and the multi-agent reinforcement learning model required by the present invention is obtained.

[0189] In step S102, the state parameters of each micro base station are input into the trained multi-agent reinforcement learning model to generate corresponding action strategies to achieve multi-service slice resource allocation.

[0190] The present invention is specifically described below in conjunction with an embodiment:

[0191] A simulation experiment was carried out according to the ultra-dense networking multi-service slice resource allocation method provided by the present invention.

[0192] There is one macro base station and five micro base stations, and the positions of the macro base station and the micro base station are fixed. Users access the base station with the largest channel gain according to their own channel gain. For the micro base station, the number of resource blocks K is set to 24, the number of users is set to 16, the bandwidth on each subcarrier B is set to 10MHz, and the transmission power on each subchannel does not exceed 1W. In the initial reinforcement learning model, the learning rate of the actor is 0.02, and the learning rate of the critic is 0.01.

[0193] In the simulation experiment, the user benefit is explained by changing the number of resource blocks. Specifically, ten steps are taken as one round, and the average of the ten steps is taken as the experimental result. Figure 4(a) to Figure 4(c) As shown in the figure, the first 50 rounds are the pre-training stage. It can be seen that all base stations try to change their strategies to obtain higher benefits until all base stations converge. When the number of resource blocks increases, the benefits of all micro base stations are improved, but the convergence speed will slow down. Among them, the number of resource blocks K in Figure 4 (a) is 20, the number of resource blocks K in Figure 4 (b) is 24; the number of resource blocks K in Figure 4 (c) is 28.

[0194] like Figure 5 As shown in Figure 2, the average user benefit can be explained by changing the cross-layer interference price. According to the game theory results, when other parameters remain unchanged, the cross-layer interference price and micro base station benefits are When allocating other resources at the same time, the average revenue of micro base stations gradually decreases with the increase of cross-layer interference price, but the reduction rate will gradually decrease. At the same time, the results of the MADDPG algorithm are compared with the theoretical values ​​of the Stackelberg game. Due to the adaptive adjustment mechanism of revenue, the results of the MADDPG algorithm are close to the game theory value and slightly higher than the game theory value.

[0195] The present invention also provides a computer-readable storage medium on which a computer program is stored, and the program implements the steps of the method when executed by a processor.

[0196] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0197] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0198] In summary, the present invention provides a method and device for allocating multi-service slice resources in an ultra-dense networking. By obtaining a pre-trained multi-agent reinforcement learning model, the state parameters of each micro base station are input into the multi-agent reinforcement learning model, and an action strategy is generated accordingly. This reduces the competition and interference of micro base stations while ensuring the communication quality and communication needs of users, thereby optimizing wireless resource allocation and alleviating spectrum tension.

[0199] In the training of the multi-agent reinforcement learning model, the macro base station revenue calculation formula and the micro base station revenue calculation formula are pre-constructed, and the resource allocation problem is modeled as a non-cooperative game to obtain the transmission power equilibrium solution and the cross-layer interference price equilibrium solution; the transmission power equilibrium solution is then used to update the state space of each policy network in the multi-agent reinforcement learning model, guiding the model to update and optimize in the preset direction, simplifying the model's computational complexity. At the same time, the multi-agent reinforcement learning model deploys a policy network and a value network on each micro base station. The value network can obtain global information and generate a more accurate estimated Q value, allowing the policy network to generate a better action strategy.

[0200] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0201] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0202] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for allocating multi-service slice resources in ultra-dense networking, characterized in that: The ultra-dense network includes at least one macro base station, each macro base station is also connected to serve multiple micro base stations; users of the micro base stations reuse slice resources of the corresponding macro base stations, and the method allocates multi-service slice resources based on cross-layer interference between the micro base station and the macro base station, and same-layer interference between adjacent micro base stations; the method includes the following steps: A multi-agent reinforcement learning model is obtained, wherein the multi-agent reinforcement learning model deploys a policy network and a value network on each micro base station; each policy network constructs a state space with the transmission rate and total transmission power of each user in the corresponding single micro base station as state parameters; an associated parameter indicating whether the user in each micro base station reuses the resource block in the macro base station is obtained, and an action space is constructed with the associated parameter set of each micro base station and the predicted transmission power set of each other micro base station as action parameters; each micro base station obtains its own state parameters, selects a corresponding action according to the policy network, and the value network of each micro base station generates an estimated Q value according to the state parameters and selected action of the corresponding micro base station and the state parameters and actions of other micro base stations, and is used to update the parameters of the policy network of the corresponding micro base station; a loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization reward value as the optimization target, and the parameters of the value network are updated; until the preset performance requirements are met; Among them, in the state update process, the macro base station constructs a macro base station profit calculation formula according to the cross-layer interference price and the cross-layer interference generated by the user multiplexing the resource block in the micro base station; the micro base station constructs a micro base station profit calculation formula according to the associated parameters, the fixed bandwidth length of the resource block, the signal to interference plus noise ratio, the same-layer interference price, the same-layer interference, the cross-layer interference price and the cross-layer interference; the macro base station is taken as the leader and each micro base station is taken as the follower to construct a non-cooperative game; the value of the associated parameter is fixed, and the reverse induction method is used to solve the micro base station profit calculation formula to obtain the transmission power equilibrium solution of each micro base station, which is used to update the state space of each strategy network; the transmission power equilibrium solution is substituted into the macro base station profit calculation formula to obtain the cross-layer interference price equilibrium solution; The state parameters of each micro base station are input into the multi-agent reinforcement learning model to generate corresponding action strategies to achieve multi-service slice resource allocation.

2. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 1, characterized in that: The macro base station constructs a macro base station revenue calculation formula according to the cross-layer interference price and the cross-layer interference generated by the user multiplexing resource blocks in the micro base station. The macro base station revenue calculation formula is: Among them, U MBS represents the macro base station revenue; U UE represents the user set of all micro base stations; U PRB Indicates the total number of resource blocks; U BS represents the set of the macro base station and all micro base stations; represents the cross-layer interference price of user i using resource block j in micro base station b; It represents the cross-layer interference caused by user i using resource block j in micro base station b.

3. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 1, characterized in that: The micro base station constructs a micro base station revenue calculation formula according to the association parameters, the fixed bandwidth length of the resource block, the signal to interference plus noise ratio, the same-layer interference price, the same-layer interference, the cross-layer interference price and the cross-layer interference. The micro base station revenue calculation formula is: st Among them, U b represents the micro base station revenue; U UE,b represents the user set of micro base station b; U s Indicates the type of slice; represents the association between user i and slice s, resource block j, and micro base station b; B represents the fixed bandwidth length of the resource block; represents the signal to interference plus noise ratio caused by user i using resource block j in micro base station b; represents the same-layer interference price of user i using resource block j at micro base station b; represents the intra-layer interference caused by user i using resource block j in micro base station b; represents the cross-layer interference price of user i using resource block j in micro base station b; represents the cross-layer interference caused by user i using resource block j in micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; U UE represents the user set of all micro base stations; U BS represents the set of the macro base station and all micro base stations; I max Indicates the maximum interference value; U PRB represents the total number of resource blocks; τ represents the total number of resource blocks.

4. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 1, characterized in that: Each strategy network constructs a state space using the transmission rate and total transmit power of each user in a single micro base station as state parameters, wherein the total transmit power uses the transmit power equalization solution, and the state parameters are expressed as: Among them, s j (t) represents the state parameters of the micro base station at time t; r N,j (t) represents the transmission rate of the Nth user multiplexing resource block j at time t; The transmission power of user i allocated to resource block j at micro base station b*; U j represents the set of users that multiplex resource block j; The value network of each micro base station generates an estimated Q value according to the state parameters and selected actions of the corresponding micro base station and the state parameters and actions of other micro base stations. The state parameters of the value network are expressed as: s j '(t)=(s j (t),a j (t),s -j (t),a -j (t)); Among them, s j (t) represents the state parameter of the micro base station at time t; a j (t) represents the action parameters of the micro base station at time t; s -j (t) represents the state parameter set of other micro base stations at time t; a -j (t) represents the action parameter set of other micro base stations at time t.

5. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 1, characterized in that: Obtain association parameters for indicating whether users in each micro base station reuse resource blocks in the macro base station, and construct an action space using the association parameter set of each micro base station and the predicted transmit power set of other micro base stations as action parameters, where the action parameters are expressed as: a j (t)={W j ,P -j }; in, Among them, a j (t) represents the action parameters of the micro base station at time t; W j Represents a collection of associated parameters; P -j represents a set of predicted transmit powers of other micro base stations; represents the association relationship between user i and slice s, resource block j, and micro base station b; represents the transmission power of user i allocated to resource block j at micro base station b; U j represents the set of users that multiplex resource block j.

6. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 1, characterized in that: The value network of each micro base station generates an estimated Q value according to the state parameters and selected actions of the corresponding micro base station and the state parameters and actions of other micro base stations, and constructs a policy gradient according to the estimated Q value, and is used to update the parameters of the policy network of the corresponding micro base station. The calculation formula of the policy gradient is: in, represents the policy gradient; θ represents the policy parameter; J(u j ) represents the cumulative estimated reward value; D represents the experience replay pool; u j (a j |s j ) represents the action strategy made by the micro base station according to the status; Represents the value network; j Indicates the state of the micro base station estimated by the value network; a j represents the action of the micro base station estimated by the value network; other Indicates the status of other micro base stations estimated by the value network; a other Represents the actions of other micro base stations predicted by the value network.

7. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 1, characterized in that: The loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization reward value as the optimization goal, and the parameters of the value network are updated. The calculation formula of the loss function is: in, represents the loss function; θ represents the strategy parameter; u j represents the adaptive weight parameter; r represents the updated result of the action; Represents the value network; j Indicates the state of the micro base station estimated by the value network; a j represents the action of the micro base station estimated by the value network; other Indicates the status of other micro base stations estimated by the value network; a ohter Indicates actions of other micro base stations estimated by the value network; represents the actual Q value.

8. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 7, characterized in that: The loss function of the estimated Q value and the actual Q value of the model is constructed with the maximization of the reward value as the optimization goal, and the parameters of the value network are updated. The calculation formula of the reward value is: Among them, reward j represents the reward value; u j represents the adaptive weight parameter; r j represents the total transmission rate of each user multiplexing resource block j of the micro base station; N represents the total number of users; r -j Indicates the total transmission rate of other micro base stations; represents the same-layer interference price; Indicates same-layer interference; represents the cross-layer interference price; represents cross-layer interference; U j represents the set of users that multiplex resource block j.

9. The method for allocating resources for multi-service slices in ultra-dense networking according to claim 8, characterized in that: The adaptive weight parameters are learned based on the state of the global environment; When u j =1, the reward value is only related to the transmission rate of the micro base station itself, which is a zero-sum game; When 0 j <1, the reward value is not only related to the transmission rate of the micro base station itself, but also to the transmission rates of other micro base stations, forming a mixed game.​ 10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images