Resource allocation method based on deep learning and related device

Abstract

One or more embodiments of the present application provide a deep learning-based resource allocation method and related equipment. The method comprises: determining a physical resource block allocation scheme of a first terminal; the physical resource block allocation scheme comprises a required number of physical resource blocks of the first terminal; and determining a resource allocation scheme of a second terminal based on the physical resource block allocation scheme. The technical solution of the present application improves the practicability of the spectrum multiplexing scheme, and further improves the wireless resource spectrum utilization rate.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present application relate to the technical field of communication, and in particular, to a resource allocation method and related equipment. BACKGROUND

[0002] In recent years, with the increasing maturity of deep learning technology and deep network technology, it is more and more applied in various fields. At present, the deep learning theory has been widely applied in the field of electric power application, including using deep learning technology to optimize the reuse scheme of limited wireless spectrum resources.

[0003] According to the reuse of wireless spectrum resources in the electric power intelligent network, the terminal can be divided into a reuse terminal and a reused terminal. Among them, the reuse terminal represents a terminal that needs to reuse the spectrum resources of other terminals, and the reused terminal represents a terminal whose spectrum resources need to be reused by other terminals.

[0004] In the related art, when using deep learning technology to allocate reuse resources, only the case that each reused terminal occupies only one spectrum resource is considered, and the case that one reused terminal occupies multiple spectrum resources is not considered. SUMMARY

[0005] Therefore, the purpose of one or more embodiments of the present application is to propose a resource allocation method based on deep learning and related equipment to solve the problems in the background art.

[0006] To achieve the above purpose, one or more embodiments of the present application provide a resource allocation method based on deep learning. The method comprises:

[0007] determining a physical resource block allocation scheme of a first terminal; the physical resource block allocation scheme comprises a required number of physical resource blocks of the first terminal;

[0008] determining a resource allocation scheme of a second terminal based on the physical resource block allocation scheme.

[0009] Optionally, the first terminal comprises a bandwidth requirement terminal and a latency requirement terminal; the bandwidth requirement terminal represents a terminal with bandwidth requirements, and the latency requirement terminal represents a terminal with latency requirements.

[0010] Optionally, the determining a physical resource block reuse scheme of a second terminal based on the physical resource block allocation scheme comprises:

[0011] determining a first terminal rate of the first terminal and a second terminal rate of the second terminal based on the physical resource block allocation scheme, wherein the first terminal has a fixed transmit power.

[0012] Based on the first terminal rate and the second terminal rate, the optimal solution of the objective function is solved to determine the resource allocation scheme of the second terminal; the optimal solution should satisfy preset constraints.

[0013] The objective function is:

[0014]

[0015] r i This indicates the terminal rate r of the terminal requiring bandwidth. j This indicates the terminal rate of the terminal required to reduce the latency, x. k,n Indicates the reuse selection of the physical resource block, r k This indicates the second terminal rate of the second terminal;

[0016] The constraints include:

[0017]

[0018] x k,n A value of 0 indicates that the second terminal does not reuse physical resource blocks, while a value of 1 indicates that it reuses physical resource blocks of terminals with bandwidth or latency requirements. This represents the signal-to-noise ratio of the physical resource block of the terminal with the bandwidth requirement. This indicates the minimum signal-to-noise ratio of the terminal physical resource block required for the bandwidth. This represents the signal-to-noise ratio of the physical resource block of the terminal, indicating the latency requirement. This represents the minimum signal-to-noise ratio of the physical resource block of the terminal required to achieve the aforementioned latency. This represents the signal-to-noise ratio of the physical resource block of the second terminal. p represents the minimum signal-to-noise ratio of the physical resource block of the second terminal. k This indicates the power of the second terminal. This represents the maximum power of the second terminal, v. i This indicates the terminal speed required by the bandwidth requirement. τ represents the minimum terminal rate required by the bandwidth requirement. j This indicates the latency requirement of the terminal. This indicates the maximum latency required by the terminal.

[0019] Optionally, the formula for calculating the terminal rate of the bandwidth-required terminal is as follows: This indicates the number of physical resource blocks required by the terminal with the bandwidth requirement. This represents the bandwidth of the nth physical resource block. a signal-to-noise ratio of a physical resource block of the bandwidth requirement terminal when the physical resource block is multiplexed, and a terminal rate calculation formula of the latency requirement terminal is a number of physical resource blocks required by the latency requirement terminal, a signal-to-noise ratio of a physical resource block of the latency requirement terminal when the physical resource block is multiplexed, and a calculation formula of the second terminal number is a signal-to-noise ratio when the second terminal multiplexes any physical resource block, p i a transmit power of the bandwidth requirement terminal, g i,B a channel gain of the bandwidth requirement terminal to a base station, p j a transmit power of the latency requirement terminal, g j,B a channel gain of the latency requirement terminal to a base station, p k a transmit power of the second terminal, g k,B a channel gain of the second terminal k to a base station, k' represents other second terminals k' multiplexing the same resource block as the second terminal k, p' represents a transmit power of the second terminal k', g k′,B a channel gain of the second terminal k' to a base station, and σ represents a noise power.

[0020] Optionally, an optimal solution of the objective function is solved by a multiplexing scheme decision model; the multiplexing scheme decision model includes a policy approximation neural network corresponding to the second terminal and a value evaluation neural network corresponding to the first terminal and the second terminal respectively;

[0021] The training step of the multiplexing scheme decision model includes:

[0022] training data is acquired, and a decision process model is constructed; the decision process model includes a state space and an action space a signal-to-noise ratio of a physical resource block of the bandwidth requirement terminal at time t, a signal-to-noise ratio of a physical resource block of the latency bandwidth requirement terminal at time t, a signal-to-noise ratio of a physical resource block of the second terminal at time t, RB k a physical resource block multiplexing selection of the second terminal, p k,l a power level selection of the second terminal, L represents the highest power level, and l represents the selected power level;

[0023] network parameters of the policy approximation neural network and the value evaluation neural network are initialized;

[0024] The following operations are performed iteratively until a preset condition is met:

[0025] Based on the current excitation value r t State value S t The current policy action is obtained through the policy approximation neural network.

[0026] The error calculation formula δ=r t +γV(s t+1 )-V(s t ), calculate the timing differential error of the bandwidth-required terminal respectively. The latency requirement is related to the timing differential error of the terminal. Timing differential error with the second terminal r t V(s) represents the excitation value at time t, γ represents the discount factor, and its value ranges from 0 to 1. t+1 ) represents state s t+1 The evaluation value, V(s) t ) represents state s t Evaluation value;

[0027] Based on the timing differential error of the bandwidth-required terminal, the timing differential error of the latency-required terminal, and the timing differential error of the second terminal, the total timing differential error is calculated. β and μ represent adjustment coefficients, ranging from 0 to 1;

[0028] Based on the timing difference error of the broadband requirement terminal, the timing difference error of the delay requirement terminal, and the timing difference error of the second terminal, the network parameters of the value evaluation neural networks corresponding to the bandwidth requirement terminal, the delay requirement terminal, and the second terminal are updated respectively.

[0029] Based on the total temporal difference error, update the network parameters of the policy approximation neural network;

[0030] Wherein, the excitation value r t Based on the excitation values ​​of the second terminal and the first terminal, the excitation function of the second terminal is determined as follows: When the first terminal is a broadband-required terminal, the excitation function is: When the first terminal is a delay-required terminal, the excitation function is:

[0031] Optionally, updating the network parameters of the value evaluation neural networks corresponding to the bandwidth-required terminal, the latency-required terminal, and the second terminal based on the timing difference error of the bandwidth-required terminal, the timing difference error of the latency-required terminal, and the timing difference error of the second terminal, respectively, includes:

[0032] updating network parameters of a value evaluation neural network corresponding to the bandwidth requirement terminal based on a timing difference error of the bandwidth requirement terminal denote a value evaluation parameter at time t, denote a learning rate, denote an output value of the value evaluation neural network at state S t .

[0033] updating network parameters of a value evaluation neural network corresponding to the bandwidth requirement terminal based on a timing difference error of the bandwidth requirement terminal denote a value evaluation parameter at time t, denote a learning rate, denote an output value of the value evaluation neural network at state S t .

[0034] updating network parameters of a value evaluation neural network corresponding to the second terminal based on a timing difference error of the second terminal denote a value evaluation parameter at time t, denote a learning rate, denote an output value of the value evaluation neural network at state S t .

[0035] Optionally, updating network parameters of the policy approximation neural network based on the total timing difference error, comprising:

[0036] updating network parameters of the policy approximation neural network based on the total timing difference error denote network parameters of a policy approximation neural network of the second terminal k at time t, t denote a policy action at time t, S t denote an output value of the policy approximation neural network, a θ denote a learning rate, denote a current policy function.

[0037] Based on the same inventive concept, one or more embodiments of the present application also provide a resource allocation apparatus based on deep learning, comprising:

[0038] a first calculation module configured to determine a physical resource block allocation scheme of a first terminal; the physical resource block allocation scheme comprises a number of physical resource blocks required by the first terminal;

[0039] a second calculation module configured to determine a resource allocation scheme of a second terminal based on the physical resource block allocation scheme.

[0040] Based on the same inventive concept, one or more embodiments of the present application further provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the deep learning based resource allocation method according to any one of the above embodiments when executing the program.

[0041] Based on the same inventive concept, one or more embodiments of the present application further provide a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the deep learning based resource allocation method according to any one of the above embodiments.

[0042] As can be seen from the above, the deep learning based resource allocation method provided by one or more embodiments of the present application determines a physical resource block allocation scheme of a first terminal; the wireless time-frequency resource allocation scheme comprises a number of physical resource blocks required by the first terminal; and determines a resource allocation scheme of a second terminal based on the physical resource block wireless time-frequency resource allocation scheme. The technical solution of the present application considers the case where the frequency spectrum resources of the multiplexed terminal are multiple, and determines the multiplexed wireless time-frequency resource and power allocation scheme of the multiplexed terminal. The technical solution of the present application improves the practicability of the frequency spectrum multiplexing scheme.

[0043] The deep learning based resource allocation apparatus, electronic device and computer readable storage medium provided by the present application can all implement the steps of the deep learning based resource allocation method described above, and therefore also have the beneficial effects of the deep learning based resource allocation method described above. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of one or more embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only one or more embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0045] Figure 1 Flowchart of the deep learning based resource allocation method of one or more embodiments of the present application;

[0046] Figure 2 Structure diagram of the deep learning based resource allocation apparatus of one or more embodiments of the present application;

[0047] Figure 3 Structure diagram of the multiplexing scheme decision model of one or more embodiments of the present application;

[0048] Figure 4A structural schematic diagram of a multiplexing scheme decision model for one or more embodiments of the present application;

[0049] Figure 5 An algorithm flowchart of a resource allocation method for one or more embodiments of the present application;

[0050] Figure 6 An electronic device hardware structural schematic diagram for one or more embodiments of the present application. DETAILED DESCRIPTION

[0051] For the purpose of making the purpose, technical scheme and advantages of the present disclosure clearer, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the drawings.

[0052] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood as the usual meaning understood by those skilled in the art to which the present disclosure belongs. The terms "first", "second", and similar terms used in one or more embodiments of the present application do not represent any order, quantity, or importance, but are only used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects listed before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0053] As described in the background section, in the related art, when allocating multiplexing resources using deep learning technology, only the case where each multiplexed terminal occupies only one frequency spectrum resource is considered, and the case where one multiplexed terminal occupies multiple frequency spectrum resources is not considered.

[0054] However, according to the actual situation of the performance requirements of power services, typical power grid services in the smart grid scenario are divided into three categories: bandwidth requirement terminals, latency requirement terminals, and information collection terminals. Among them, the latency requirement terminal represents a terminal that has requirements for latency, such as power grid control type service terminals; the bandwidth requirement terminal represents a terminal that has requirements for bandwidth, such as mobile application type terminals. In combination with the actual situation, information collection type terminals, such as smart meters and telemetry terminals, often have low requirements for latency, bandwidth, and the like. In order to ensure communication quality, bandwidth requirement terminals and latency requirement terminals often need multiple frequency spectrum resources. In this case, the multiplexing wireless time-frequency resource allocation and signal interference in the communication network become complex. In the related art, the multiplexing wireless time-frequency resource allocation scheme is not considered when the terminal uses multiple frequency spectrum resources.

[0055] Therefore, the application proposes a resource allocation method based on deep learning to solve the above problems.

[0056] Before detailing the application, first introduce the concept of physical resource block. 12 subcarriers are continuous in frequency, and one slot in time domain, which is called 1 RB. According to a subcarrier bandwidth of 15k, the bandwidth of 1 RB is 180kHz.

[0057] Reference Figure 1 The deep learning-based resource allocation method of one or more embodiments of the application includes the following steps:

[0058] Step S101: Determine the physical resource block allocation scheme of the first terminal; The physical resource block allocation scheme includes the number of physical resource blocks required by the first terminal.

[0059] Since the multiplexing scheme of physical resource blocks needs to be formulated based on its usage, it is necessary to first determine the physical resource block allocation scheme of the first terminal.

[0060] As mentioned above, in the power network, some terminals have higher requirements for bandwidth or delay, while some terminals have lower requirements for bandwidth and delay. Therefore, in the technical solution of the application, high requirement terminals are used as multiplexed terminals, and low requirement terminals are used as multiplexed terminals.

[0061] In combination with the actual situation, in some embodiments, the high requirement terminal (i.e. multiplexed terminal) can include bandwidth requirement terminal and delay requirement terminal, and the low requirement terminal (i.e. multiplexed terminal) can include information acquisition terminal. That is, the first terminal includes bandwidth requirement terminal and delay requirement terminal, and the second terminal includes information acquisition terminal.

[0062] For convenience of description, in this specification, all physical resource blocks in a base station coverage scenario are represented as N={1, 2,..., N}, bandwidth requirement terminals are represented as i∈I={1, 2,..., I}, delay requirement terminals are represented as j∈J={1, 2,..., J}, and information acquisition terminals are represented as K={1, 2,..., K}.

[0063] It can be understood that if at least one mutually orthogonal physical resource block resource is allocated to all first terminals, and the second terminal multiplexes the physical resource block resource of the first terminal, there is no interference between the first terminals at this time, and the channel gain can be represented as: g i,B =h i,B β i,B , where h i,B represents the small-scale fading between terminal i and base station B, and β i,B represents the large-scale fading between terminal i and base station B.

[0064] The above wireless time-frequency resource allocation scheme includes the number of resources occupied by each bandwidth requirement terminal and each delay requirement terminal. In the application scenario described above, the signal-to-noise ratio of the bandwidth requirement terminal i on a single physical resource block is: p i denotes the transmission power of the bandwidth requirement terminal, g i,B denotes the channel gain from the bandwidth requirement terminal to the base station, σ 2 denotes the noise power. At this time, the rate of the bandwidth requirement terminal on a single physical resource block can be represented as B RB denotes the bandwidth of a single physical resource block. Assuming that the minimum rate required by the bandwidth requirement terminal is then the minimum requirement of the bandwidth requirement terminal for physical resource blocks is:

[0065] For the delay requirement terminal, under the condition of delay constraint slicing, assuming that the minimum rate required by the delay requirement terminal is the maximum delay requirement allowed is then the average delay calculation formula of the delay requirement terminal is: λ j denotes the arrival rate of the data packet of the delay requirement terminal, L j denotes the length of the data packet. The arrival rate and the length are obtained based on a queuing model. At this time, the minimum requirement of the delay requirement terminal for physical resource blocks is: denotes the rate of the delay requirement terminal on a single physical resource block to the base station.

[0066] Step S102: Based on the above physical resource block allocation scheme, determine the resource allocation scheme of the second terminal.

[0067] According to the physical resource block allocation scheme of the first terminal determined in step S101, when the second terminal multiplexes the resources of the bandwidth requirement terminal, the signal-to-noise ratio of a single physical resource block of the bandwidth requirement terminal is: p k denotes the transmission power of the second terminal, g k,B denotes the channel gain from the second terminal to the base station, σ 2 denotes the noise power.

[0068] When the second terminal multiplexes the resources of the delay requirement terminal, the signal-to-noise ratio of a single physical resource block of the delay requirement terminal is: p j denotes the transmission power of the delay requirement terminal, g j,B denotes the channel gain from the delay requirement terminal to the base station, σ 2denotes the noise power.

[0069] Considering that the second terminals reuse a single physical resource block, multiple second terminals can reuse one physical resource block, and the signal-to-noise ratio of the second terminals can be considered as: p i denotes the transmit power of the bandwidth requirement terminal, g i,B denotes the channel gain from the bandwidth requirement terminal to the base station, p j denotes the transmit power of the delay requirement terminal, g j,B denotes the channel gain from the delay requirement terminal to the base station, p k denotes the transmit power of the second terminal, g k,B denotes the channel gain from the second terminal k to the base station, k' denotes other second terminals k' that reuse the same resource block as the second terminal k, p' denotes the transmit power of the second terminal k', g k' denotes the transmit power of the second terminal k', g k′,B denotes the channel gain from the second terminal k' to the base station, σ 2 denotes the noise power.

[0070] Thus, the rate of the bandwidth requirement terminal can be expressed as: denotes the number of physical resource blocks required by the bandwidth requirement terminal, denotes the bandwidth on the nth physical resource block, denotes the signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal when the physical resource block is reused, and the terminal rate calculation formula of the delay requirement terminal is denotes the number of physical resource blocks required by the delay requirement terminal, denotes the signal-to-noise ratio of the physical resource block of the delay requirement terminal when the physical resource block is reused, and the calculation formula of the number of second terminals is denotes the signal-to-noise ratio when the second terminal reuses any physical resource block, p i denotes the transmit power of the bandwidth requirement terminal, g i,B denotes the channel gain from the bandwidth requirement terminal to the base station, p j denotes the transmit power of the delay requirement terminal, g j,B denotes the channel gain from the delay requirement terminal to the base station, p k denotes the transmit power of the second terminal k, g k,B denotes the channel gain from the second terminal k to the base station, k' denotes other second terminals k' that reuse the same resource block as the second terminal k, p' denotes the transmit power of the second terminal k', g k′,B denotes the channel gain from the second terminal k' to the base station, σ2 Indicates noise power.

[0071] In some embodiments of the technical solution of this application, using To find the resource reuse scheme for the objective function, r i This indicates the terminal's bandwidth requirement, r. j The terminal rate, x, represents the latency requirement of the terminal. k,n This represents the selection of physical resource blocks for reuse, and rk represents the second terminal rate. This resource reuse scheme aims to maximize system capacity by optimizing the selection of reused resources and power levels of the second terminal while ensuring the needs of the first terminal are met.

[0072] When solving the above objective function, the following constraints also need to be satisfied:

[0073]

[0074] x k,n A value of 0 indicates that the second terminal does not reuse physical resource blocks, while a value of 1 indicates that it reuses physical resource blocks of terminals with bandwidth or latency requirements. This represents the signal-to-noise ratio of the physical resource blocks of the terminal that require bandwidth. This represents the minimum signal-to-noise ratio of the terminal's physical resource block required for bandwidth. This represents the signal-to-noise ratio of the physical resource blocks of the terminal, indicating the latency requirements. This represents the minimum signal-to-noise ratio of the physical resource blocks required by the terminal to achieve latency. This represents the signal-to-noise ratio of the physical resource blocks of the second terminal. p represents the minimum signal-to-noise ratio of the physical resource block of the second terminal. k This indicates the transmission power of the second terminal. This represents the maximum power of the second terminal, v. i This indicates the terminal's bandwidth requirement and terminal speed. τ represents the minimum terminal rate required by the bandwidth-restricted terminal. j This indicates the latency requirement for the terminal. This indicates the maximum latency required by the terminal.

[0075] Among them, the first constraint means that the second terminal can only select one physical resource block for multiplexing from either the bandwidth-required terminal or the latency-required terminal; the second constraint is the minimum rate limit for the bandwidth-required terminal; the third constraint is the maximum latency requirement for the latency-required terminal; the fourth constraint is the minimum signal-to-noise ratio requirement for each physical resource block; and the fifth constraint is the maximum power level limit for the second terminal.

[0076] In some embodiments, the optimal solution of the objective function can be solved by a multiplexing scheme decision model.

[0077] A reinforcement learning task can generally be described using a Markov Decision Process (MDP), specifically: and is in an environment, each state is and its perception of the current environment; the machine can only affect the environment through actions, when the machine performs an action, the environment will be transferred to another state with a certain probability; at the same time, the environment will feedback a reward to and according to a potential reward function.

[0078] The present application can use the MDP model to model the environment to which the technical solution is applied. The MDP model defines a tuple (S, A, P, R), where S represents the state space, A represents the action space, P represents the state transition probability, and R represents the reward function.

[0079] In the present application, the state space can be represented as: represents the signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal at time t, represents the signal-to-noise ratio of the physical resource block of the delay bandwidth requirement terminal at time t; the action space can be represented as: represents the signal-to-noise ratio of the physical resource block of the second terminal at time t, RB k represents the physical resource block multiplexing selection of the second terminal, p k,l represents the power level selection of the second terminal, L represents the highest power level, and l represents the selected power level.

[0080] In order to better perform the process of iterative update, as shown in Figure 3 The present application proposes a policy approximation neural network and a value approximation neural network, wherein the policy approximation neural network is used to generate the policy at the current time.

[0081] Since the optimization goal is to maximize the system capacity, if the sum of the capacities of the first terminal and the second terminal is used as the incentive value, since the capacity value of the first terminal is greater than that of the second terminal, the above method will result in a result more favorable to the first terminal, and the influence of the second terminal on the system capacity is weakened, resulting in that the system capacity optimization is not obvious, and the multiplexing scheme decision model is not easy to converge.

[0082] Therefore, the present application proposes an improved multi-agent A2C algorithm, as shown in Figure 4As shown, a critic neural network is allocated to the bandwidth requirement terminal as a whole, a critic neural network is allocated to the delay requirement terminal as a whole, and a separate critic neural network and a policy approximation neural network are allocated to each second terminal, so that the number of policy approximation neural networks and critic neural networks is not necessarily equal, thereby being more conducive to centralized learning.

[0083] The policy approximation neural network parameters can be optimized based on a total temporal difference (TD) error, and each critic neural network can be optimized based on a TD error of the corresponding terminal.

[0084] The total TD error is calculated by the formula where β and μ respectively represent adjustment coefficients, and take values of 0 to 1, represents a TD error value of the second terminal, represents a TD error value of the bandwidth requirement terminal, represents a TD error value of the delay requirement terminal.

[0085] The TD error value of the second terminal can be calculated by the TD error calculation formula δ = r t + γV(s t+1 )- V(s t ), respectively. The TD error of the bandwidth requirement terminal The TD error of the delay requirement terminal r t represents an incentive value at time t, γ represents a discount factor, and takes a value of 0 to 1, V(s t+1 ) represents an evaluation value of state s t+1 , and V(s t ) represents an evaluation value of state st.

[0086] Thus, the training process of the multiplexing scheme decision model can be considered as:

[0087] Data for training is obtained, and a decision process model is constructed; the above-mentioned decision process model includes a state space and an action space represents a signal-to-noise ratio of a physical resource block of the bandwidth requirement terminal at time t, represents a signal-to-noise ratio of a physical resource block of the delay bandwidth requirement terminal at time t, RB k represents a physical resource block multiplexing selection of the second terminal, p k,l represents a power level selection of the second terminal, L represents the highest power level, and l represents the selected power level;

[0088] Initialize network parameters of the policy approximation neural network and the value evaluation neural network;

[0089] Iteratively perform the following operations until a preset condition is met:

[0090] Based on the incentive value r t , the state value S t , and the current policy action obtained through the policy approximation neural network;

[0091] Calculate the timing difference error of the bandwidth requirement terminal t , the timing difference error of the latency requirement terminal t+1 , and the timing difference error of the second terminal t through the error calculation formula δ = r t + γV(s t+1 t) - V(s t+1 t-1), respectively. r t represents the incentive value at time t, γ represents a discount factor, and takes a value of 0 to 1, V(s t+1 t) represents the evaluation value of state s t+1 , and V(s t t-1) represents the evaluation value of state s t .

[0092] Based on the timing difference error of the bandwidth requirement terminal, the timing difference error of the latency requirement terminal, and the timing difference error of the second terminal, calculate the total timing difference error β and μ represent adjustment coefficients, and take values of 0 to 1.

[0093] Based on the timing difference error of the bandwidth requirement terminal, the timing difference error of the latency requirement terminal, and the timing difference error of the second terminal, update the network parameters of the value evaluation neural networks corresponding to the bandwidth requirement terminal, the latency requirement terminal, and the second terminal, respectively.

[0094] Based on the total timing difference error, update the network parameters of the policy approximation neural network.

[0095] wherein the incentive value r t is determined based on the incentive value of the second terminal and the incentive value of the first terminal, the incentive function of the second terminal is the incentive function when the first terminal is a bandwidth requirement terminal is the incentive function when the first terminal is a latency requirement terminal is

[0096] updating network parameters of a value evaluation neural network corresponding to the bandwidth requirement terminal based on the timing difference error of the bandwidth requirement terminal

[0097] updating network parameters of a value evaluation neural network corresponding to the bandwidth requirement terminal based on the timing difference error of the bandwidth requirement terminal denotes a value evaluation parameter at time t, denotes a learning rate, denotes an output value of the value evaluation neural network in state s t at time t;

[0098] updating network parameters of a value evaluation neural network corresponding to the bandwidth requirement terminal based on the timing difference error of the bandwidth requirement terminal denotes a value evaluation parameter at time t, denotes a learning rate, denotes an output value of the value evaluation neural network in state s t at time t;

[0099] updating network parameters of a value evaluation neural network corresponding to the bandwidth requirement terminal based on the timing difference error of the bandwidth requirement terminal denotes a value evaluation parameter at time t, denotes a learning rate, denotes an output value of the value evaluation neural network in state s t at time t.

[0100] updating network parameters of a policy approximation neural network based on the total timing difference error, comprising:

[0101] updating network parameters of a policy approximation neural network based on the total timing difference error denotes network parameters of a policy approximation neural network of the second terminal k at time t, t denotes a policy action at time t, t denotes an output value of the policy approximation neural network, θ denotes a learning rate, denotes a current policy function.

[0102] In some embodiments, the above content can be represented by a pseudo code as shown in Figure 5

[0103] ​The technical solution provided in this application solves the problem of wireless resource reuse in high-requirement terminals, realizes high-requirement wireless resource reuse, accelerates the convergence speed of learning algorithms, and improves the utilization rate of wireless resource spectrum.

[0104] It is understandable that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0105] It should be noted that the methods of one or more embodiments of this application can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this application, and the multiple devices will interact with each other to complete the method described.

[0106] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a resource allocation device based on deep learning.

[0108] like Figure 2 As shown, a deep learning-based resource allocation device includes:

[0109] The first calculation module 11 is configured to determine the physical resource block allocation scheme of the first terminal; the wireless time and frequency resource allocation scheme includes the number of physical resource blocks required by the first terminal;

[0110] The second calculation module 12 is configured to determine the resource allocation scheme of the second terminal based on the wireless time-frequency resource allocation scheme of the physical resource block.

[0111] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, when implementing one or more embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0112] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0113] Figure 6 A more specific electronic device hardware structure diagram provided by the embodiment is shown, which can include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected through the bus 1050 for communication between each other inside the device.

[0114] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0115] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1020 and called and executed by the processor 1010.

[0116] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0117] The communication interface 1040 is used to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0118] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0119] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary to implement the embodiments of the present application, and does not have to contain all the components shown in the figure.

[0120] The electronic device of the above embodiment is used to implement the corresponding method in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0121] The computer readable medium of the embodiment includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0122] Those skilled in the art should understand that the above discussion of any embodiment is only exemplary and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present disclosure, the above embodiments or technical features in different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of one or more embodiments of the present application as described above. In order to be brief, they are not provided in detail.

[0123] Additionally, to simplify the description and discussion, and so as not to obscure the embodiments of the application(s) being presented, the well-known functions or constructions of integrated circuit (IC) chips and other components can or can not be shown in the figures and will not be discussed in detail. Moreover, the apparatus can be shown in block diagram form in order to avoid obscuring the embodiments of the application(s) being presented, and this also acknowledges the fact that the details in regard to the implementation of such block diagram apparatus are highly dependent on the platform within which the application(s) of the present application is to be implemented (i.e., such details should be well within the purview of one of ordinary skill in the art to design the specific implementation of the block diagram apparatus). Where specific details are set forth in order to describe an illustrative embodiment of the disclosure, it will be apparent to one of ordinary skill in the art that the application(s) of the present application can be practiced without these specific details. In other instances, detailed descriptions of well-known methods, procedures, components, circuits and the like can omit so as not to obscure the disclosure of the application(s) of the present application. The description thus should be regarded as illustrative and not restrictive.

[0124] Although the present disclosure has been described in conjunction with the specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0125] It is therefore intended to embrace all such alternatives, modifications and variations in the spirit and principle of the application(s) of the present application. Accordingly, any and all departures from the above described embodiments are included within the scope of the present disclosure.

Claims

1. A resource allocation method characterized by, The method comprises: determining a physical resource block allocation scheme of a first terminal; the physical resource block allocation scheme comprises a required number of physical resource blocks of the first terminal; the first terminal comprises a bandwidth requirement terminal and a latency requirement terminal; the bandwidth requirement terminal represents a terminal with bandwidth requirements, and the latency requirement terminal represents a terminal with latency requirements; based on the physical resource block allocation scheme, determining a resource allocation scheme of a second terminal, comprising: based on the physical resource block allocation scheme, determining a first terminal rate of the first terminal and a second terminal rate of the second terminal; based on the first terminal rate and the second terminal rate, solving an optimal solution of an objective function to determine the resource allocation scheme of the second terminal; the optimal solution should satisfy a preset constraint condition; wherein the objective function is: ； a terminal rate indicating the bandwidth requirement terminal, a terminal rate indicating the latency requirement terminal, a multiplexing selection indicating the physical resource block, a second terminal rate indicating the second terminal, the constraint condition comprises: ； ； ； ； ； 0 indicates that the second terminal does not reuse the physical resource block, and 1 indicates that the physical resource block of the bandwidth requirement terminal or the time delay requirement terminal is reused, a signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal, a minimum signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal, a signal-to-noise ratio of the physical resource block of the time delay requirement terminal, a minimum signal-to-noise ratio of the physical resource block of the time delay requirement terminal, a signal-to-noise ratio of the physical resource block of the second terminal, a minimum signal-to-noise ratio of the physical resource block of the second terminal, a transmission power of the second terminal, a maximum power of the second terminal, a terminal rate of the bandwidth requirement terminal, a minimum terminal rate of the bandwidth requirement terminal, a time delay of the time delay requirement terminal, a maximum time delay of the time delay requirement terminal.

2. The method of claim 1, wherein, The terminal rate calculation formula of the bandwidth requirement terminal is , represents the number of physical resource blocks required by the bandwidth requirement terminal, represents the bandwidth of one physical resource block, represents the signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal when the physical resource block is multiplexed, and the terminal rate calculation formula of the delay requirement terminal is , represents the number of physical resource blocks required by the delay requirement terminal, represents the signal-to-noise ratio of the physical resource block of the delay requirement terminal when the physical resource block is multiplexed, and the calculation formula of the second terminal number is , represents the signal-to-noise ratio when the second terminal multiplexes any physical resource block, , represents the transmission power of the bandwidth requirement terminal, represents the channel gain of the bandwidth requirement terminal to the base station, represents the transmission power of the delay requirement terminal, represents the channel gain of the delay requirement terminal to the base station, represents the transmission power of the second terminal , represents the channel gain of the second terminal to the base station, represents other second terminals multiplexing the same resource block as the second terminal , represents the transmission power of the second terminal , represents the channel gain of the second terminal to the base station, represents the noise power.

3. The method according to claim 1 or 2, characterized in that, solving the optimal solution of the objective function through a multiplexing scheme decision model; the multiplexing scheme decision model comprises a policy approximation neural network corresponding to the second terminal and a value evaluation neural network corresponding to the first terminal and the second terminal respectively; the training steps of the multiplexing scheme decision model comprise: Obtaining training data, and constructing a decision process model; the decision process model includes state space and action space , representing the signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal at time, representing the signal-to-noise ratio of the physical resource block of the delay bandwidth requirement terminal at time, representing the physical resource block multiplexing selection of the second terminal, representing the power level selection of the second terminal, , representing the highest power level, representing the selected power level; initializing network parameters of the policy approximation neural network and the value evaluation neural network; iteratively performing the following operations until a preset condition is met: Incentive value based on current time State value A current policy action is obtained through the policy approximation neural network. Through the error calculation formula Calculate the timing differential error of the bandwidth-required terminal respectively. The latency requirement is the timing differential error of the terminal. Timing differential error with the second terminal ; express The incentive value at any given moment. This represents the discount factor, which ranges from 0 to 1. Representing state The evaluation value, Representing state Evaluation value; calculating a total timing difference error based on the timing difference error of the bandwidth requirement terminal, the timing difference error of the latency requirement terminal and the timing difference error of the second terminal , denotes an adjustment coefficient, and takes a value from 0 to 1; updating network parameters of the value evaluation neural network corresponding to the bandwidth requirement terminal, the latency requirement terminal and the second terminal respectively based on a timing difference error of the bandwidth requirement terminal, a timing difference error of the latency requirement terminal and a timing difference error of the second terminal; updating network parameters of the policy approximation neural network based on the total timing difference error.

4. The method of claim 3, wherein, The updating of the network parameters of the value evaluation neural network corresponding to the bandwidth requirement terminal, the latency requirement terminal and the second terminal respectively based on the timing difference error of the bandwidth requirement terminal, the timing difference error of the latency requirement terminal and the timing difference error of the second terminal comprises: updating network parameters of the value evaluation neural network corresponding to the bandwidth requirement terminal based on a timing difference error of the bandwidth requirement terminal ; denotes a value evaluation parameter at time t, denotes a learning rate, denotes an output of the value evaluation neural network in state s updating network parameters of the value evaluation neural network corresponding to the latency requirement terminal based on a timing difference error of the latency requirement terminal ; denotes a value evaluation parameter at time t, denotes a learning rate, denotes an output of the value evaluation neural network in state at time t. updating network parameters of the second terminal's corresponding value evaluation neural network based on timing difference errors of the second terminal ; denotes a value evaluation parameter at time t, denotes a learning rate, denotes an output value of the value evaluation neural network in state at time t.

5. The method of claim 3, wherein, The updating of the network parameters of the policy approximation neural network based on the total timing difference error comprises: updating network parameters of the policy approximation neural network based on the total timing difference error . denotes network parameters of the policy approximation neural network of the second terminal k at the time instant, denotes the policy action at the time instant, denotes the output value of the policy approximation neural network, denotes a learning rate, denotes the current policy function.

6. A resource allocation apparatus characterized by comprising: The method comprises: a first calculation module configured to determine a physical resource block allocation scheme of a first terminal; the physical resource block allocation scheme comprises a required number of physical resource blocks of the first terminal; the first terminal comprises a bandwidth requirement terminal and a latency requirement terminal; the bandwidth requirement terminal represents a terminal with bandwidth requirements, and the latency requirement terminal represents a terminal with latency requirements; a second module configured to determine a resource allocation scheme of a second terminal based on the physical resource block allocation scheme, comprising: based on the physical resource block allocation scheme, determining a first terminal rate of the first terminal and a second terminal rate of the second terminal; based on the first terminal rate and the second terminal rate, solving an optimal solution of an objective function to determine the resource allocation scheme of the second terminal; the optimal solution should satisfy a preset constraint condition; wherein the objective function is: ； a terminal rate of the bandwidth requirement terminal, a terminal rate of the latency requirement terminal, a multiplexing selection of the physical resource block, a second terminal rate of the second terminal; the constraint condition comprises: ； ； ； ； ； 0 indicates that the second terminal does not reuse the physical resource block, and 1 indicates that the physical resource block of the bandwidth requirement terminal or the time delay requirement terminal is reused, a signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal, a minimum signal-to-noise ratio of the physical resource block of the bandwidth requirement terminal, a signal-to-noise ratio of the physical resource block of the time delay requirement terminal, a minimum signal-to-noise ratio of the physical resource block of the time delay requirement terminal, a signal-to-noise ratio of the physical resource block of the second terminal, a minimum signal-to-noise ratio of the physical resource block of the second terminal, a transmission power of the second terminal, a maximum power of the second terminal, a terminal rate of the bandwidth requirement terminal, a minimum terminal rate of the bandwidth requirement terminal, a time delay of the time delay requirement terminal, a maximum time delay of the time delay requirement terminal.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 5.

Images