Multi-agent resource allocation method and related equipment

Abstract

The invention provides a multi-agent resource allocation method. The method comprises the following steps: setting a training round, firstly initializing state information, setting an excitation value according to an optimized objective function, constructing a network environment model and a resource allocation model based on multiple pieces of action information, and obtaining initial action information of multiple agents. Wherein the network environment model is configured to change the network environment where the intelligent agents are located in response to the multiple pieces of action information, and each intelligent agent comprises a resource allocation model. And obtaining state information based on the network environment model, inputting the state information into the resource allocation model to enable the resource allocation model to optimize the action information, and adjusting parameters of the network environment model to obtain optimized action information. And obtaining state information based on the optimized action information. And performing resource allocation based on the state information and the target function. A plurality of agents can automatically optimize and allocate resources, and the efficiency of a resource allocation algorithm is improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-agent deep reinforcement learning, and in particular to a multi-agent resource allocation method and related equipment. Background Art

[0002] In recent years, with the gradual increase in wireless terminal devices, and the limited wireless spectrum resources, in some application scenarios where wireless devices are densely populated, the wireless spectrum resources are usually reused to access too many wireless devices. However, due to mutual interference between wireless devices, it is necessary to allocate the wireless spectrum resources reused by wireless devices to ensure the communication quality of more devices.

[0003] For example, with regard to the problem of wireless resource allocation in power heterogeneous network scenarios, authorized frequency band resources are becoming increasingly scarce. In order to ensure the communication needs of power services, it is necessary to consider the reuse of authorized and unlicensed spectrum resources by multi-standard communication terminals in power industries. In addition, in actual typical power business scenarios such as substations and converter stations, WAPI (WLAN Authentication and Privacy Infrastructure) communication access has been carried out in power terminal communications. However, there is currently little research on the reuse of authorized and unlicensed frequency bands by power terminals. Therefore, it is necessary to conduct research on wireless resource reuse in related scenarios.

[0004] At present, the optimization methods for spectrum resource allocation of wireless devices include convex relaxation traditional algorithms and machine learning algorithms. Compared with traditional algorithms, machine learning has great advantages in solving complex problems. Machine learning algorithms use neural network models to perform deep learning optimization on allocation results. When using traditional resource allocation methods to allocate intelligent agent resources with multiple communication mode selections, resource selections, and power selections, there are problems such as a large number of overall parameters, high algorithm complexity, and slow convergence, which leads to poor optimization efficiency of the algorithm. Therefore, a method for intelligent agent resource allocation that can reduce the number of parameters and improve optimization efficiency is needed. Summary of the invention

[0005] In view of this, the purpose of this application is to propose a multi-agent resource allocation method and related equipment.

[0006] Based on the above purpose, the present application provides a multi-agent resource allocation method, including: setting a training round, first initializing the state information, setting the reward function according to the optimized objective function, calculating the incentive value based on the reward function, constructing a network environment model and a resource allocation model based on the action information of multiple agents, and obtaining the action information of multiple agents in the initial state. Among them, the network environment model is configured to change the network environment in which the agent is located in response to multiple action information, and each agent includes a resource allocation model. The resource allocation model includes a value network and a policy network. Based on the network environment model, the state information is obtained, the state information is input into the resource allocation model, the policy network outputs the optimized action information, the value network evaluates the state information and outputs the evaluation value, and the time difference error is calculated based on the evaluation value and the reward function. Based on the time difference error, the parameters of the policy network and the parameters of the value network are optimized, so that the current strategy is optimized, and the action information of the agent is updated to the next state; repeat the above cycle steps until the state information is consistent with the objective function, and the resources are allocated according to the optimized strategy.

[0007] In some embodiments, the state information is obtained based on the network environment model, the state information is input into the resource allocation model, the policy network outputs the optimized action information, the value network evaluates the state information and outputs the evaluation value, and the time series difference error is calculated based on the evaluation value and the incentive value, which specifically includes: inputting the state information into the policy network, the policy network selects the action information according to the current policy, outputs the optimized action information and calculates the signal-to-noise ratio and rate corresponding to the optimized action information. The incentive value is calculated based on the signal-to-noise ratio and the rate, and the action information is input into the value network so that the value network evaluates based on the current state information and the state information at the next moment to obtain the evaluation value. The time series difference error is calculated based on the evaluation value and the incentive value, and the parameters of the policy network are updated based on the time series difference error.

[0008] In some embodiments, the method for calculating the incentive value specifically includes: in response to the state information satisfying the first or second constraint function, the agent obtains a positive incentive. In response to the state information failing to satisfy the first or second constraint function, the agent obtains a negative incentive. Based on the evaluation value and the incentive value, the weight value of the action information is adjusted through the policy network, specifically including: in response to the state information approaching the target function, increasing the weight of the current action information. In response to the state information being away from the target function, reducing the weight of the current action information.

[0009] In some embodiments, the action information includes: the working mode, resource block position and power level of the agent. The agent can switch between at least two working modes, two resource block positions and two power levels. The output layer of the strategy network also includes a multi-head action network model, which includes a mode action layer, a resource block action layer and a power action layer. The parameters of the strategy network are updated based on the time difference error, specifically including: based on the weight value of the action information, the actions included in the action information are weighted and calculated, and the probability distribution of the working mode, resource block position and power level are calculated respectively. Based on the weight value, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block position, and the power action layer calculates the power probability distribution of the power level. Based on the mode probability distribution, the mode action layer outputs the working mode with the highest probability as the optimized working mode. Based on the resource block probability distribution, the resource block action layer outputs the resource block position with the highest probability as the optimized resource block position. Based on the power probability distribution, the power action layer outputs the power level with the highest probability as the optimized power level. Based on the optimized action mode, optimized resource block location and optimized power level, the multi-head action network model outputs the optimized action information.

[0010] In some embodiments, the state information is input into the policy network, the policy network selects action information according to the current policy, outputs the optimized action information and calculates the signal-to-noise ratio and rate corresponding to the optimized action information, specifically including: comparing the state information with the objective function. In response to the state information fully satisfying the objective function, the agent resource allocation is completed. In response to the state information not fully satisfying the objective function, the value network and the policy network parameters are adjusted based on the reward function, and the parameters of the resource allocation model are updated. The optimized action information is used as the action information, and the state information is used as the state information, and is input into the resource allocation model again, and the action information is optimized until the state information output by the resource allocation model fully satisfies the objective function.

[0011] In some embodiments, the network environment model further includes a base station, an access point and a plurality of cellular terminals. The cellular terminal connects to the base station through the authorized frequency band for communication, and the power of the cellular terminal signal received by the base station is the first receiving power. In response to the action information and the optimization action information, some agents connect to the authorized frequency band to communicate with the base station, and the other agents connect to the unauthorized frequency band to communicate with the access point. The network environment model also includes environmental noise, and the power of environmental noise is noise power. The state information includes: the first signal-to-noise ratio of the cellular terminal connecting to the authorized frequency band, the agent's authorized and unauthorized frequency band selection, the second signal-to-noise ratio of the agent connecting to the authorized frequency band, the first average rate of the agent connecting to the unauthorized frequency band, and the number of users of the agent connecting to the unauthorized frequency band. Based on the optimization action information, the parameters of the network environment model are adjusted to obtain the state information, which specifically includes: in response to the agent connecting to the authorized frequency band, the agent reuses the resource block position occupied by the cellular terminal, the power of the agent signal received by the base station from the reused resource block is the first reuse power, and the sum of the powers of other agent signals received by the base station from the reused resource block is the second receiving power. In response to the agent connecting to the unlicensed frequency band, multiple agents access the unlicensed frequency band through competition, and the power of the agent signal received by the access point is the third receiving power. A first signal-to-noise ratio is calculated based on the first receiving power, the first multiplexing power, and the noise power. A second signal-to-noise ratio is calculated based on the first receiving power, the first multiplexing power, the second receiving power, and the noise power. A third signal-to-noise ratio of the agent connecting to the unlicensed frequency band is calculated based on the third receiving power and the noise power, and a first average rate is calculated based on the third signal-to-noise ratio. State information is obtained based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the first average rate.

[0012] In some embodiments, the minimum signal-to-noise ratio of the cellular terminal connecting to the base station is set as the first signal-to-noise ratio threshold, the minimum signal-to-noise ratio of the intelligent agent connecting to the base station is set as the second signal-to-noise ratio threshold, and the minimum rate of the intelligent agent connecting to the access point is set as the first average rate threshold. The objective function is to maximize the total throughput of the system To optimize the target, solve the resource reuse scheme, where X represents the mode selection of the agent, F represents the channel selection of the agent, and P represents the power selection of the agent. is the throughput of the agent in the unlicensed band, is the throughput of the agent in the authorized frequency band. The objective function also includes a first constraint function and a second constraint function. The first constraint function includes: the first signal-to-noise ratio is greater than the first signal-to-noise ratio threshold, and the second signal-to-noise ratio is greater than the second signal-to-noise ratio threshold. The second constraint function includes: the first average rate is greater than the first average rate threshold. The incentive value is calculated based on the signal-to-noise ratio and the rate, and the action information is input into the value network so that the value network is evaluated based on the current state information and the state information at the next moment to obtain an evaluation value, which specifically includes: in response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive incentive value. In response to at least one of the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtains a negative incentive value. The temporal difference error is calculated based on the evaluation value of the positive incentive value or the negative incentive value for the current state information, and the parameters of the strategy network and the value network are optimized based on the temporal difference error, and the above steps are repeated.

[0013] Based on the same inventive concept, an embodiment of the present application also provides a multi-agent resource allocation device, including: a modeling module, used to obtain action information of multiple agents, construct a network environment model and a resource allocation model based on the multiple action information, and set an objective function; wherein the network environment model changes the network environment in which the agents are located in response to the action information of the multiple agents, and each of the agents includes a resource allocation model; an optimization module, used to obtain state information based on the network environment model, input the state information into the resource allocation model, and the resource allocation model optimizes the action information to obtain optimized action information; an adjustment module, used to adjust the parameters of the network environment model based on the optimized action information to obtain state information; an allocation module, used to allocate resources based on the state information and the objective function.

[0014] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a multi-agent resource allocation method as described in any one of the above.

[0015] Based on the same inventive concept, an embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any one of the multi-agent resource allocation methods described above.

[0016] As can be seen from the above, the multi-agent resource allocation method provided by the present application enables multiple agents to optimize their own actions by setting a resource allocation model for each agent, thereby reducing the amount of optimization calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0019] Figure 2 A flowchart of a multi-agent resource allocation method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a resource allocation model provided in an embodiment of the present application;

[0021] Figure 4 A flowchart of a method for obtaining optimization action information provided by another embodiment of the present application;

[0022] Figure 5 A flowchart of a method for obtaining optimization action information based on a weight value provided by another embodiment of the present application;

[0023] Figure 6 A schematic diagram of converting an output layer into a multi-head action network model provided in an embodiment of the present application;

[0024] Figure 7 A flowchart of a method for determining whether resource allocation is completed provided by another embodiment of the present application;

[0025] Figure 8 A flowchart of a method for obtaining status information provided by another embodiment of the present application;

[0026] Fig. 9 A flowchart of a method for determining resource allocation incentives for agents connected to different channels provided by another embodiment of the present application;

[0027] Fig.10 A schematic diagram of a resource allocation model provided by another embodiment of the present application;

[0028] Fig.11 A schematic diagram of the structure of a multi-agent resource allocation device provided in an embodiment of the present application;

[0029] Fig.12 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0031] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0032] In related technologies, a neural network model is usually used to allocate spectrum resources for wireless devices. However, as the number of wireless devices increases, the number of parameters in the traditional neural network model also increases. Due to limited computing resources, it is increasingly difficult for the computing server to process the increasing number of model parameters. Therefore, it is necessary to design a wireless device spectrum resource allocation algorithm that can reduce the complexity of the algorithm, so as to reduce the complexity of the algorithm and reduce the computing power consumption during the optimization process.

[0033] In view of this, the present invention provides a multi-agent resource allocation method, which sets a resource allocation model for multiple agents that need to allocate resources, so that each agent can judge whether the current action information meets the objective function based on its own observation results, and optimize its own action, and finally obtain action information that meets the requirements of the objective function. By splitting the complex algorithm and allocating it to multiple agents to optimize the action information by themselves, the amount of model parameters can be reduced.

[0034] refer to Figure 1, which is a schematic diagram of an application scenario of the text correction method provided in an embodiment of the present application. The application scenario includes a base station 101, an access point 201, multiple cellular terminals 102 and multiple agents 202. Among them, all cellular 102 terminals communicate with the base station 101 through the authorized frequency band 301, and the agents 202 can switch between the authorized frequency band 301 and the unlicensed frequency band 302. In response to the allocation result, some agents 202 communicate with the base station 101 through the authorized frequency band 301, and another part of the agents 202 communicate with the access point 201 through the unlicensed frequency band 302, and at the same time, the agents 202 can only communicate through one of the authorized frequency band 301 and the unlicensed frequency band 302. Among them, there is co-channel interference 303 between devices communicating through the authorized frequency band 301.

[0035] The base station can provide cellular network signal access for the intelligent body and the cellular terminal. The type of cellular network is not limited and can be reasonably selected according to actual application needs. For example, it can be a combination of one or more of the GSM network, CDMA network, 3G, 4G, and 5G networks.

[0036] The multi-agent resource allocation method provided in the embodiments of the present application can be applied to different types of intelligent agent system resource allocation scenarios. For example, the intelligent agent can be power equipment in the power grid, urban road traffic equipment, urban public safety equipment, intelligent environmental monitoring equipment, etc.

[0037] The multi-agent resource allocation method of the embodiment of the present application can be applied to scenarios such as power equipment, urban road traffic equipment, urban public safety equipment, and intelligent environmental monitoring equipment. The allocation target can be adjusted based on the signal strength requirements of the agents in different application scenarios to adapt to the allocation of agent resources in different application scenarios.

[0038] Combine the following Figure 1 The multi-agent resource allocation method and related devices according to the exemplary implementation of the present application are described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0039] See also Figure 2 The embodiment of the present invention provides a multi-agent resource allocation method, comprising the following steps:

[0040] S1, set up a training round, first initialize the state information, set the incentive value according to the optimized objective function, obtain the initial action information of multiple agents, and build a network environment model and a resource allocation model based on the multiple action information. The network environment model is configured to change the network environment of the agent in response to the multiple action information, and each agent includes a resource allocation model.

[0041] In this embodiment, the intelligent agent is an intelligent device with certain computing capabilities and can send and receive wireless signals. Based on the adjustment of the intelligent agent's action information, the strength, frequency and connected signal source of the signal sent by the intelligent agent can be changed. Figure 1 The system shown includes multiple intelligent agents, each of which includes a resource allocation model, which is used to adjust the action information of the intelligent agent according to the current state information of the intelligent agent fed back in the network environment model. The multiple intelligent agents adjust their own action information multiple times, with the optimization goal that all the intelligent agents included in the network environment model can satisfy the objective function, thereby realizing wireless resource allocation for all the intelligent agents in the network environment model.

[0042] Among them, the network environment model can be a mathematical model or a real network environment. When the network environment model is a mathematical model, the action information can be configured to the real network environment after the mathematical model is optimized, thereby reducing the impact on the network environment; when the network environment model is a real network environment, the optimization results of the resource allocation model can be directly applied to the real network environment, thereby improving the accuracy of the optimization results.

[0043] S2, obtains state information based on the network environment model, inputs the state information into the resource allocation model, the policy network outputs the action, the value network evaluates the current state and outputs the evaluation value, and calculates the temporal difference error according to the evaluation value and the reward function.

[0044] The state information of the agent changes with the action information of the agent. The type of state information is not limited and can be reasonably selected according to the actual application needs. For example, the state information may include one or more of the interference to the agent, the signal-to-noise ratio when the agent transmits and receives signals, and the connection rate. The optimization action information includes the same type of information as the action information.

[0045] S3, optimizes the strategy network parameters and value network parameters through temporal difference error, so that the current strategy is optimized and the agent action is updated to the next state.

[0046] The status information is of the same type as the information included in the status information.

[0047] S4, repeat the above cycle steps until the state information is consistent with the objective function, and allocate resources according to the optimized strategy.

[0048] Among them, the repeated steps are to set the training round, set the incentive value according to the optimized objective function, the agent interacts with the environment through the current strategy, the agent selects the action and the incentive value obtained in combination with the current strategy, the value network evaluates the action state and calculates the evaluation value, calculate the temporal difference error according to the evaluation value and the reward function, optimize the strategy network parameters and the value network parameters through the temporal difference error, so that the current strategy is optimized, the agent action is updated to the next state, and the above steps are repeated until the training reaches the optimal resource allocation of the objective function.

[0049] In some embodiments, Figure 3 As shown, the resource allocation model also includes a value network and a strategy network.

[0050] Among them, the resource allocation model optimizes the action information of the agent based on the A2C (actor-critic) algorithm. Specifically, the actor-critic method (Actor-Critic) consists of two artificial neural networks. The first network is the policy network (Actor), which is used to approximate the policy π, and the second network (Critic) is the value network, which is used to approximate the value function. The value network approximates to provide the value estimate of the state and the estimated value of the subsequent state, thereby guiding the learning of the actor and further optimizing the policy network. Figure 2 The structure and general information flow of an Actor-Critic based agent is shown. The policy network takes in a state and outputs a probability distribution over actions in that state. The value network takes in the state and the reward and is used for policy learning.

[0051] Specifically, the network environment model (environment) responds to the action information output by the strategy network at the previous moment and outputs the state information S t With motivational information t . The value network obtains state information S t With motivational information t , output evaluation information V(S t ). The policy network responds to the state information S t and evaluation information V(S t ), output optimization action information A t The network environment model (environment) responds to the optimization action information A output by the policy network t Output status information S again t+1 With motivational information t+1 , and repeat the above process until the state information converges to the optimal state.

[0052] like Figure 4As shown, step S2 specifically includes:

[0053] S21, input the state information into the policy network, the policy network selects an action according to the current policy probability, outputs the action and calculates the corresponding signal-to-noise ratio and rate.

[0054] S22, calculating the incentive value according to the above signal-to-noise ratio and rate, and inputting the above action into the value network so that the value network evaluates the action based on the current state and next state information to obtain the evaluation value.

[0055] Among them, the value network is used to evaluate the action information and output the corresponding evaluation value. By inputting the action information into the value network and obtaining the corresponding evaluation value, the evaluation value is output according to the current value network parameters.

[0056] S23, based on the above evaluation value and incentive value, the timing difference error is calculated by the TD error formula, which is used to update the strategic network mode layer, resource layer and power layer parameters.

[0057] Among them, the parameters of the value network and the strategy network are adjusted based on the calculation of the temporal difference error. The temporal difference error is calculated by the following formula:

[0058]

[0059] in, represents the incentive value given by the reward function to agent k at the t+1 state, γ represents the discount factor, ranging from 0 to 1, (the discount factor parameter determines the importance of future rewards. A larger discount factor will make the agent pay more attention to future rewards, while a smaller discount factor will make the agent pay more attention to immediate rewards, generally 0.95), Represents the state information s of the value network for agent k t The evaluation value of Represents the state information s of the value network for agent k t+1 The evaluation value.

[0060] In some embodiments, the action information includes: the working mode, resource block location and power level of the agent, wherein the agent can switch between at least two working modes, two resource block locations and two power levels.

[0061] In this embodiment, the action information of the agent is expressed by the following formula:

[0062]

[0063] Among them, X d is the X working mode selection of agent d; RB dFor agent d, I resource block position selection, 12 consecutive subcarriers in frequency and 1 slot in time domain are called a resource block RB. According to the bandwidth of 1 subcarrier is 15k, it can be concluded that the bandwidth of 1 resource block is 180kHz; P d,l The power level of the agent d is determined by its own factory parameters. The agent discretizes the power into L levels, which are determined by the following formula:

[0064]

[0065] The maximum power of the wireless signal transmitted by the agent.

[0066] The output layer of the strategy network also includes a multi-head action network model, which includes a mode action layer, a resource block action layer and a power action layer.

[0067] Among them, Figure 6 As shown in the figure, the output layer of the strategy network is replaced by the traditional output layer including action 1, action 2, action 3...action n, with a multi-head action network model including a mode action layer, a resource block action layer and a power action layer, and the mode action layer output includes X 1 , X 2 …X n The action mode of the resource block action layer output includes RB 1 , RB 2 …RB F The resource block location, the power action layer output includes P 1 , P 2 …P L power level, thereby reducing the number of parameters output by the policy network.

[0068] Specifically, in the output layer of the neural network model, the parameter with the highest probability is usually output in the form of probability distribution. However, if the three parameters in the action information of the intelligent agent are combined and output, the number of units output by the output layer is N×F×L. Taking N=2, F=50, and L=10 as an example, the number of units in the output layer is 1000. If the hidden layer before the output layer includes 200 units, then the number of weights in the last layer alone is equal to 200×1000=200000 (ignoring the bias term). The output of the multi-head action network model can calculate the weights of the three parameters in the action information separately, and the number of units in the output layer changes from N×F×L to N+F+L. When N=2, F=50, and L=10, the number of units in the output layer is 2+50+10=62. When the number of units in the hidden layer before the output layer is 200, the number of weights output is 200×62=12400, and the number of parameters to be trained is reduced by about 16 times.

[0069] like Figure 5 As shown, step S3 specifically includes:

[0070] S31, based on the weight value of the action information, weighted calculation is performed on the actions included in the action information, and probability distribution of the working mode, resource block position and power level are calculated respectively.

[0071] S32, based on the weight value, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block position, and the power action layer calculates the power probability distribution of the power level.

[0072] Among them, Figure 6 As shown, the mode action layer, resource block action layer and power action layer calculate the mode probability distribution, resource block probability distribution and power probability distribution respectively, and based on the weight value, increase the selection probability of the action mode, resource block position and power level included in the action information corresponding to the state information that meets the objective function in the multi-head action network model, and reduce the selection probability of the action mode, resource block position and power level included in the action information corresponding to the state information that does not meet the objective function in the multi-head action network model.

[0073] S33, based on the mode probability distribution, the mode action layer outputs the working mode with the highest probability as the optimized working mode. Based on the resource block probability distribution, the resource block action layer outputs the resource block position with the highest probability as the optimized resource block position. Based on the power probability distribution, the power action layer outputs the power level with the highest probability as the optimized power level.

[0074] S34, based on the optimized action mode, the optimized resource block position and the optimized power level, the multi-head action network model outputs the optimized action information.

[0075] Among them, the working mode, resource block position and probability level with the highest probability are selected and output, and combined as optimization action information.

[0076] The resource allocation method provided in the present application sets a strategy network and a value network for each intelligent agent. The input layer of the value network includes state information. The value network calculates and outputs the evaluation value through the hidden layer. The strategy network takes the state information and the evaluation value as input, and sets the output layer as a multi-head action network model to output the probability distribution of various action information.

[0077] In some embodiments, Figure 3 As shown, the resource allocation model also includes a reward function;

[0078] like Figure 7 As shown, step S4 specifically includes:

[0079] S41, compare the state information with the target function.

[0080] Among them, by setting the training round, first initialize the state information, set the incentive value according to the optimized objective function, the agent interacts with the environment through the current strategy, the agent selects the action and the incentive value obtained by combining the strategy probability, the value network evaluates the action state and calculates the evaluation value, calculates the temporal difference error according to the evaluation value and the reward function, optimizes the policy network parameters and the value network parameters through the temporal difference error, so that the current strategy is optimized, the agent action is updated to the next state, and the above cycle steps are repeated until the training converges.

[0081] By comparing the state information with the objective function, it is determined whether the state information meets the optimization goal set in the objective function, and the action information corresponding to the state information that meets the objective function and the action information corresponding to the state information that does not meet the objective function are processed separately.

[0082] S42, in response to the state information completely satisfying the objective function, the agent resource allocation is completed.

[0083] Among them, according to the set training round, when the state information fully meets the convergence state, the agent resource allocation is completed. When the state information cannot fully meet the training round or convergence state, the value network and policy network parameters are adjusted based on the reward function, and the parameters of the resource allocation model are updated.

[0084] The reward function is calculated by the following formula:

[0085] When the agent accesses the authorized frequency band, the reward function is set to

[0086]

[0087] When the agent accesses the unlicensed frequency band, the reward function is set to

[0088]

[0089] When the state information satisfies the first constraint function, the incentive value is set. It is used to indicate the incentive value of the intelligent agent to connect to the authorized frequency band. When the state information satisfies the second constraint function, the incentive value is set. It is used to indicate the incentive value when the agent connects to the unlicensed frequency band. When the state information cannot satisfy the first constraint function or the second constraint function, a penalty value is set. The penalty values ​​are constants -h L and -h U ,According to the different impacts on user capacity in authorized and unauthorized modes, different penalty constants are set to accelerate convergence.

[0090] The parameters of the value network and the policy network are adjusted based on the calculation of the temporal difference error. The temporal difference error is calculated by the following formula:

[0091]

[0092] in, represents the incentive value given by the reward function to agent k at the t+1 state, γ represents the discount factor, ranging from 0 to 1, (the discount factor parameter determines the importance of future rewards. A larger discount factor will make the agent pay more attention to future rewards, while a smaller discount factor will make the agent pay more attention to immediate rewards, generally 0.95), Represents the state information s of the value network for agent k t The evaluation value of Represents the state information s of the value network for agent k t+1 The evaluation value.

[0093] Adjust the parameters of the value network based on the following formula:

[0094]

[0095] in, represents the value network parameter of agent k at the time of obtaining state information, Represents the learning rate of the value network. The value range is usually between 0 and 1. A higher learning rate means that new information has a greater impact on the network and the update speed is faster, while a lower learning rate means that the update speed is slower and more stable. It is generally set to around 0.001; Represents the policy gradient of the current value network; Indicates that the status information s t The evaluation value output by the value network after being input into the value network of agent k.

[0096] Adjust the parameters of the mode selection layer in the policy network based on the following formula:

[0097]

[0098] in, represents the policy network parameters of the agent k-mode selection layer at time t; represents the action mode output by the policy network at time t; s t Represents the output value of the above strategy approximating the neural network; α θ represents the policy network learning rate; π θ Indicates the current strategy; Represents the policy gradient of the current policy network.

[0099] Adjust the parameters of the power selection layer in the strategy network based on the following formula:

[0100]

[0101] in, represents the policy network parameters of the power selection layer of agent k at time t; Represents the power level of the strategy network output at time t.

[0102] Adjust the parameters of the resource selection layer in the policy network based on the following formula:

[0103]

[0104] in, represents the policy network parameters of the resource selection layer of agent k at time t; Represents the power level of the strategy network output at time t.

[0105] S43, taking the optimized action information as the action information and the state information as the state information, and inputting them into the resource allocation model again, and optimizing the action information until the state information output by the resource allocation model fully satisfies the objective function.

[0106] Among them, by taking the optimized action information as the action information and the state information as the state information, inputting them into the resource allocation model again, re-optimizing the action information, and repeating the optimization process until the optimization is completed, the different action information of the agent can be evaluated, and finally the action information whose corresponding state information meets the objective function can be selected. And by performing the same operation on multiple agents, the allocation of wireless resources to all agents can be achieved.

[0107] In some embodiments, Figure 1 As shown, the network environment model also includes a base station, an access point and multiple cellular terminals. The cellular terminal connects to the base station through the authorized frequency band for communication, and the power of the cellular terminal signal received by the base station is the first receiving power. In response to the action information and the optimized action information, some intelligent agents connect to the authorized frequency band to communicate with the base station, and the other part of the intelligent agents connect to the unauthorized frequency band to communicate with the access point. The network environment model also includes environmental noise, and the power of environmental noise is noise power. The status information includes: the first signal-to-noise ratio of the cellular terminal connected to the authorized frequency band, the second signal-to-noise ratio of the intelligent agent connected to the authorized frequency band, and the first average rate of the intelligent agent connected to the unauthorized frequency band.

[0108] In this embodiment, the number of subchannels in the licensed frequency band is recorded as N, expressed as N = {1, ..., n}, where n is one subchannel in all channels; the number of all cellular terminals is recorded as C, expressed as C = {1, ..., c}, where d is one of the cellular terminals; the number of all agents is recorded as D, expressed as D = {1, ..., d}, where d is one of the agents; the total bandwidth of the unlicensed frequency band is recorded as B. Among them, the environmental noise includes mutual interference between agents, between cellular devices, and between agents and cellular devices, as well as interference from other wireless signals in the area, and noise caused by phenomena such as signal attenuation, network congestion, and equipment failure.

[0109] like Figure 8 As shown, step S21 specifically includes:

[0110] S211, in response to the agent connecting to the authorized frequency band, the agent reuses the resource block position occupied by the cellular terminal, the power of the agent signal received by the base station in the reused resource block is the first multiplexing power, and the sum of the powers of the agent signals received by the base station in other multiplexing resource blocks is the second receiving power.

[0111] S212, in response to the agent connecting to the unlicensed frequency band, multiple agents access the unlicensed frequency band through competition, and the power of the agent signal received by the access point is a third receiving power.

[0112] S213: Calculate a first signal-to-noise ratio based on the first received power, the first multiplexing power, and the noise power.

[0113] Here, take the case where all cellular terminals are connected to the base station in advance through the authorized frequency band and there is no co-frequency interference between the cellular terminals as an example:

[0114] The first signal-to-noise ratio can be calculated by the following formula:

[0115]

[0116] in, Represents the binary coefficient of the cellular terminal on channel resource n. When the cellular terminal occupies channel resource n The value is 1, when the cellular terminal does not occupy the channel resource n The value of is 0; Represents the binary coefficient of the agent on channel resource n. When the agent occupies channel resource n The value is 1 when the agent has no channel resource n The value of p is 0; c Indicates the wireless signal transmission power of the cellular terminal; g c,BS represents the channel gain of the cellular terminal and is calculated by the following formula:

[0117] gc,BS =h c,BS β c,BS

[0118] h c,BS represents the small-scale fading between the cellular terminal and the base station, β c,BS Represents the large-scale fading between the cellular terminal and the base station.

[0119] S214: Calculate a second signal-to-noise ratio based on the first received power, the first multiplexing power, the second received power, and the noise power.

[0120] When the agent reuses channel n, multiple agents may reuse one channel, and the second signal-to-noise ratio can be calculated by the following formula:

[0121]

[0122] Where d′ is the other agent that multiplexes channel n, Represents the binary coefficient of the agent on channel resource n. When the agent occupies channel resource n The value is 1 when the agent has no channel resource n The value of p is 0; d represents the wireless signal transmission power of the intelligent agent; g d,BS It represents the channel gain of the agent connecting to the authorized frequency band, which is calculated by the following formula:

[0123] g d,BS =h d,BS β d,BS

[0124] h d,BS represents the small-scale fading between the agent and the base station, β d,BS represents the large-scale fading between the agent and the base station; N0 represents the noise power.

[0125] S215, calculating a third signal-to-noise ratio of the agent connected to the unlicensed frequency band based on the third received power and the noise power, and calculating a first average rate based on the third signal-to-noise ratio.

[0126] The third signal-to-noise ratio is calculated by the following formula:

[0127]

[0128] in, Indicates the power of the agent connecting to the unlicensed frequency band, It represents the channel gain of the agent connecting to the unlicensed frequency band, which is calculated by the following formula:

[0129]

[0130] represents the small-scale fading between the agent and the access point, Represents the large-scale fading between the agent and the access point.

[0131] The first average rate is calculated based on the third signal-to-noise ratio, using the following formula:

[0132]

[0133] Among them, B u Indicates the total bandwidth of the unlicensed frequency band. Represents the binary coefficient of the agent on the unlicensed frequency band. The value is 1 when the agent does not occupy the unlicensed frequency band The value of is 0.

[0134] S216: Acquire state information based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the first average rate.

[0135] In some embodiments, the constraint function includes a first constraint function and a second constraint function;

[0136] like Fig. 9 As shown, step S22 specifically includes:

[0137] S221, in response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive incentive value.

[0138] S222: In response to at least one of the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtains a negative incentive value.

[0139] The first constraint function is used to determine whether the agent connected to the base station through the authorized frequency band meets the resource allocation requirements, and the second constraint function is used to determine whether the agent connected to the access point through the unlicensed frequency band meets the resource allocation requirements.

[0140] S223, in response to the above incentive value, the value network calculates the evaluation values ​​of the previous state and the next state respectively, and calculates the temporal difference error according to the TD error formula, which is used to optimize the parameters of the strategy network and the value network.

[0141] All the above state information is input as state information into the resource allocation model, the optimized action information is used as action information, and the action information is optimized again.

[0142] Among them, by taking the optimized action information as the action information and the state information as the state information, inputting them into the resource allocation model again, re-optimizing the action information, and repeating the optimization process until the optimization is completed, the different action information of the intelligent agent can be evaluated, and finally the action information whose corresponding state information meets the objective function can be screened out. And by performing the above operation on each intelligent agent, the allocation of wireless resources for all intelligent agents can be realized.

[0143] In some embodiments, the minimum signal-to-noise ratio for a cellular terminal to connect to a base station is set as a first signal-to-noise ratio threshold, the minimum signal-to-noise ratio for an intelligent agent to connect to a base station is set as a second signal-to-noise ratio threshold, and the minimum rate for an intelligent agent to connect to an access point is set as a first average rate threshold. The first constraint function includes: the first signal-to-noise ratio is greater than the first signal-to-noise ratio threshold, and the second signal-to-noise ratio is greater than the second signal-to-noise ratio threshold. The second constraint function includes: the first average rate is greater than the first average rate threshold.

[0144] In this embodiment, to maximize the total system throughput As the objective function, solve the resource reuse scheme, where X represents mode selection, F represents channel selection, and P represents power selection. is the throughput of end users in the unlicensed band, is the throughput of the end user in the licensed frequency band.

[0145] When solving the above objective function, the following constraints need to be met:

[0146] The first constraint function includes:

[0147]

[0148] The second constraint function includes:

[0149]

[0150] The objective function may further include a first constraint condition, a second constraint condition and a third constraint condition.

[0151] The first constraints include:

[0152]

[0153] The second constraint includes:

[0154]

[0155] The third constraint includes:

[0156] 0≤p d ≤P max

[0157] Among them, p d represents the transmission power of the agent’s wireless signal, P max Indicates the maximum power of the wireless signal transmitted by the agent, which is determined by the agent's own performance parameters; represents the first signal-to-noise ratio threshold, It represents the second signal-to-noise ratio threshold, which can be reasonably configured according to resource allocation requirements. The first constraint condition indicates that the terminal's licensed channel and unlicensed channel mode selection has a binary value, the second constraint condition is that the terminal can only select one communication mode, and the third constraint condition is the maximum power limit of the terminal.

[0158] As a preferred embodiment, Fig.10 As shown, the multi-agent resource allocation method provided by the present application also includes a resource allocation network model as shown in the figure, wherein the strategy network (Actor network) outputs actions to change environmental parameters, and the environment outputs state information to the value network (Critic network). The value network includes an input layer, 3 hidden layers (the number of hidden layers needs to comprehensively consider multiple factors such as complexity and practical problems, and appropriate parameters can be selected) and an output layer. The state information is distributed to the 3 hidden layers through the input layer of the value network. The hidden layer performs weighted calculation on the state information and outputs the state value through the output layer as an advantage function. The advantage function and the state information of the environment output are input into the strategy network. The input layer of the strategy network distributes the advantage function and the state information to the three hidden layers of the strategy network. The three hidden layers perform weighted calculations on the state information and the function, and classify the corresponding action information through the softmax activation function. The action information of the agent is split and output through the mode action layer, resource block (RB) action layer and power action layer in the multi-head action network model (hierarchical multi-head strategy) respectively. The probability distribution of multiple mode actions, the probability distribution of multiple resource block actions and the probability distribution of multiple power actions are given. The mode action, resource block action and power action with the highest probability are used as the optimized action information, and the environment is adjusted based on the actions in the optimized action information.

[0159] As can be seen from the above embodiments of the present application, the present application sets a resource allocation model for each agent, so that multiple agents can optimize their own actions by themselves, thereby reducing the amount of optimization calculations. By configuring the resource allocation model as an A2C network model including a value network and a policy network, the state information is compared with the objective function through the value function, and the weight of the action information that meets the objective function is increased, and the weight of the action information that does not meet the objective function is reduced, so that the allocation structure of the resource allocation model can converge. Then, the output layer of the policy network is set to a multi-head structure, thereby reducing the number of weights that need to be calculated during the optimization process, accelerating the convergence of the structure, and improving the utilization of wireless resources.

[0160] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0161] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0162] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a multi-agent resource allocation device.

[0163] refer to Fig.11 , the multi-agent resource allocation device comprises:

[0164] The modeling module 501 is used to obtain the action information of multiple agents, build a network environment model and a resource allocation model based on the multiple action information, and set the objective function. The network environment model changes the network environment of the agent in response to the action information of the multiple agents, and each agent includes a resource allocation model.

[0165] The optimization module 502 is used to obtain state information based on the network environment model, input the state information into the resource allocation model, and the resource allocation model optimizes the action information to obtain optimized action information.

[0166] The adjustment module 503 is used to adjust the parameters of the network environment model based on the optimization action information to obtain state information.

[0167] The allocation module 504 is used to allocate resources based on the state information and the objective function.

[0168] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0169] The apparatus of the above-mentioned embodiment is used to implement the corresponding multi-agent resource allocation method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0170] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-agent resource allocation method described in any of the above embodiments is implemented.

[0171] Fig.12 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0172] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0173] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0174] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0175] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0176] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0177] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0178] The electronic device of the above embodiment is used to implement the corresponding multi-agent resource allocation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0179] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-agent resource allocation method described in any of the above embodiments.

[0180] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0181] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multi-agent resource allocation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0182] It should be noted that the embodiments of the present application can be further described in the following manner:

[0183] A multi-agent resource allocation method, comprising:

[0184] Set up a training round, first initialize the state information, set the reward function according to the optimized objective function, calculate the incentive value based on the reward function, build a network environment model and a resource allocation model based on the action information of multiple agents, and obtain the action information of multiple agents in the initial state. Among them, the network environment model is configured to change the network environment of the agent in response to multiple action information, and each agent includes a resource allocation model. The resource allocation model includes a value network and a policy network.

[0185] The state information is obtained based on the network environment model, and the state information is input into the resource allocation model. The policy network outputs the optimization action information. The value network evaluates the state information and outputs the evaluation value. The temporal difference error is calculated based on the evaluation value and the reward function.

[0186] Based on the temporal difference error, the parameters of the strategy network and the value network are optimized so that the current strategy is optimized and the action information of the agent is updated to the next state. The above cycle steps are repeated until the state information is consistent with the objective function, and resources are allocated according to the optimized strategy.

[0187] Optionally, the state information is obtained based on the network environment model, the state information is input into the resource allocation model, the policy network outputs the optimization action information, the value network evaluates the state information and outputs the evaluation value, and the time series difference error is calculated based on the evaluation value and the incentive value, specifically including:

[0188] The state information is input into the policy network, which selects action information according to the current policy, outputs the optimized action information and calculates the signal-to-noise ratio and rate corresponding to the optimized action information.

[0189] The incentive value is calculated based on the signal-to-noise ratio and the rate, and the action information is input into the value network so that the value network performs evaluation based on the current state information and the state information at the next moment to obtain an evaluation value.

[0190] The temporal difference error is calculated based on the evaluation value and the incentive value, and the parameters of the strategy network are updated based on the temporal difference error.

[0191] Optional, where

[0192] The calculation method of the incentive value specifically includes:

[0193] In response to the state information satisfying the first or second constraint function, the agent obtains positive incentives.

[0194] In response to the state information failing to satisfy the first or second constraint function, the agent obtains a negative incentive.

[0195] Based on the evaluation value and incentive value, the weight value of the action information is adjusted through the strategy network, including:

[0196] In response to the state information approaching the target function, the weight of the current action information is increased.

[0197] In response to the state information being far away from the objective function, the weight of the current action information is reduced.

[0198] Optional, where

[0199] The action information includes: the working mode of the agent, the location of the resource block and the power level. The agent can switch between at least two working modes, two resource block locations and two power levels.

[0200] The output layer of the strategy network also includes a multi-head action network model, which includes a mode action layer, a resource block action layer and a power action layer.

[0201] The parameters of the strategy network are updated based on the temporal difference error, including:

[0202] Based on the weight value of the action information, the actions included in the action information are weightedly calculated, and the probability distribution of the working mode, resource block position and power level are calculated respectively.

[0203] Based on the weight values, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block position, and the power action layer calculates the power probability distribution of the power level.

[0204] Based on the mode probability distribution, the mode action layer outputs the working mode with the highest probability as the optimized working mode. Based on the resource block probability distribution, the resource block action layer outputs the resource block position with the highest probability as the optimized resource block position. Based on the power probability distribution, the power action layer outputs the power level with the highest probability as the optimized power level.

[0205] Based on the optimized action mode, optimized resource block location and optimized power level, the multi-head action network model outputs the optimized action information.

[0206] Optional, where

[0207] The state information is input into the policy network, which selects action information according to the current policy, outputs the optimized action information, and calculates the signal-to-noise ratio and rate corresponding to the optimized action information, including:

[0208] Compare the state information with the target function.

[0209] In response to the state information fully satisfying the objective function, the agent resource allocation is completed.

[0210] In response to the state information not being able to fully satisfy the objective function, the value network and the policy network parameters are adjusted based on the reward function, and the parameters of the resource allocation model are updated.

[0211] The optimized action information is used as the action information, and the state information is used as the state information, which are input into the resource allocation model again, and the action information is optimized until the state information output by the resource allocation model fully satisfies the objective function.

[0212] Optional, where

[0213] The network environment model also includes a base station, an access point and a plurality of cellular terminals. The cellular terminal communicates with the base station via the authorized frequency band, and the power of the cellular terminal signal received by the base station is the first receiving power. In response to the action information and the optimized action information, some agents communicate with the base station via the authorized frequency band, and the other agents communicate with the access point via the unauthorized frequency band. The network environment model also includes environmental noise, and the power of environmental noise is the noise power.

[0214] The status information includes: the first signal-to-noise ratio of the cellular terminal connecting to the authorized frequency band, the authorized and unauthorized frequency band selection of the intelligent body, the second signal-to-noise ratio of the intelligent body connecting to the authorized frequency band, the first average rate of the intelligent body connecting to the unauthorized frequency band, and the number of users of the intelligent body connecting to the unauthorized frequency band.

[0215] Based on the optimization action information, the parameters of the network environment model are adjusted to obtain status information, including:

[0216] In response to the agent connecting to the authorized frequency band, the agent reuses the resource block position occupied by the cellular terminal, the power of the agent signal received by the base station in the reused resource block is the first reuse power, and the sum of the powers of other agent signals received by the base station in the reused resource block is the second reception power.

[0217] In response to the agent connecting to the unlicensed frequency band, multiple agents access the unlicensed frequency band through competition, and the power of the agent signal received by the access point is the third receiving power.

[0218] A first signal-to-noise ratio is calculated based on the first received power, the first multiplexing power, and the noise power.

[0219] A second signal-to-noise ratio is calculated based on the first received power, the first multiplexing power, the second received power, and the noise power.

[0220] A third signal-to-noise ratio of the agent connected to the unlicensed frequency band is calculated based on the third received power and the noise power, and a first average rate is calculated based on the third signal-to-noise ratio.

[0221] State information is acquired based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the first average rate.

[0222] Optional, where

[0223] The minimum signal-to-noise ratio for the cellular terminal to connect to the base station is set as the first signal-to-noise ratio threshold, the minimum signal-to-noise ratio for the agent to connect to the base station is set as the second signal-to-noise ratio threshold, and the minimum rate for the agent to connect to the access point is set as the first average rate threshold.

[0224] The objective function is to maximize the total system throughput To optimize the target, solve the resource reuse scheme, where X represents the mode selection of the agent, F represents the channel selection of the agent, and P represents the power selection of the agent. is the throughput of the agent in the unlicensed band, is the throughput of the agent in the licensed frequency band.

[0225] The objective function also includes a first constraint function and a second constraint function. The first constraint function includes: the first signal-to-noise ratio is greater than a first signal-to-noise ratio threshold, and the second signal-to-noise ratio is greater than a second signal-to-noise ratio threshold.

[0226] The second constraint function includes: the first average rate is greater than a first average rate threshold.

[0227] The incentive value is calculated based on the signal-to-noise ratio and the rate, and the action information is input into the value network so that the value network can evaluate based on the current state information and the state information at the next moment to obtain the evaluation value, which specifically includes:

[0228] In response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive incentive value.

[0229] In response to at least one of the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtains a negative incentive value.

[0230] The temporal difference error is calculated based on the evaluation value of the positive incentive value or the negative incentive value for the current state information, and the parameters of the strategy network and the value network are optimized based on the temporal difference error, and the above steps are repeated.

[0231] A multi-agent resource allocation device, comprising:

[0232] The modeling module is used to obtain the action information of multiple agents, build a network environment model and a resource allocation model based on the multiple action information, and set the objective function. The network environment model changes the network environment of the agent in response to the action information of the multiple agents, and each agent includes a resource allocation model.

[0233] The optimization module is used to obtain state information based on the network environment model, input the state information into the resource allocation model, and the resource allocation model optimizes the action information to obtain optimized action information.

[0234] The adjustment module is used to adjust the parameters of the network environment model based on the optimization action information and obtain the status information.

[0235] The allocation module is used to allocate resources based on state information and objective function.

[0236] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any one of the above methods.

[0237] A non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a computer to execute any of the above methods.

[0238] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0239] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0240] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0241] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-agent resource allocation method, comprising: A training round is set, firstly initializing the state information, setting the reward function according to the optimized objective function, calculating the incentive value based on the reward function, constructing a network environment model and a resource allocation model based on the action information of the plurality of agents, and obtaining the action information of the plurality of agents in the initial state; wherein the network environment model is configured to change the network environment of the agent in response to the plurality of action information, each agent includes a resource allocation model; the resource allocation model includes a value network and a policy network; The state information is acquired based on the network environment model, the state information is input into the resource allocation model, the policy network outputs the optimization action information, the value network evaluates the state information and outputs the evaluation value, and the temporal difference error is calculated based on the evaluation value and the reward function; Optimizing the parameters of the strategy network and the parameters of the value network based on the temporal difference error, so that the current strategy is optimized and the action information of the agent is updated to the next state; The above cycle steps are repeated until the state information is consistent with the objective function, and resources are allocated according to the optimized strategy.

2. The multi-agent resource allocation method according to claim 1, wherein: The state information is obtained based on the network environment model, the state information is input into the resource allocation model, the strategy network outputs the optimization action information, the value network evaluates the state information and outputs the evaluation value, and the temporal difference error is calculated based on the evaluation value and the incentive value, specifically including: Inputting the state information into the strategy network, the strategy network selecting the action information according to the current strategy, outputting the optimized action information and calculating the signal-to-noise ratio and rate corresponding to the optimized action information; Calculate the incentive value based on the signal-to-noise ratio and the rate, input the action information into the value network, so that the value network performs evaluation based on the current state information and the state information at the next moment to obtain the evaluation value; The temporal difference error is calculated based on the evaluation value and the incentive value, and the parameters of the policy network are updated based on the temporal difference error.

3. The multi-agent resource allocation method according to claim 2, wherein: The calculation method of the incentive value specifically includes: In response to the state information satisfying the first or second constraint function, the agent obtains positive incentive; In response to the state information failing to satisfy the first or second constraint function, the agent obtains a negative incentive; The step of adjusting the weight value of the action information through the strategy network based on the evaluation value and the incentive value specifically includes: In response to the state information being close to the target function, increasing the weight of the current action information; In response to the state information being far away from the objective function, the weight of the current action information is reduced.

4. The multi-agent resource allocation method according to claim 3, wherein: The action information includes: the working mode, resource block position and power level of the agent; wherein the agent can switch between at least two of the working modes, two of the resource block positions and two of the power levels; The output layer of the strategy network also includes a multi-head action network model, and the multi-head action network model includes a mode action layer, a resource block action layer and a power action layer; The updating of the parameters of the strategy network based on the temporal difference error specifically includes: Based on the weight value of the action information, weighted calculation is performed on the actions included in the action information, and probability distribution of the working mode, the resource block position and the power level are calculated respectively; Based on the weight value, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block position, and the power action layer calculates the power probability distribution of the power level; Based on the mode probability distribution, the mode action layer outputs the working mode with the highest probability as the optimized working mode; based on the resource block probability distribution, the resource block action layer outputs the resource block position with the highest probability as the optimized resource block position; based on the power probability distribution, the power action layer outputs the power level with the highest probability as the optimized power level; Based on the optimization action mode, the optimization resource block position and the optimization power level, the multi-head action network model outputs the optimization action information.

5. The multi-agent resource allocation method according to claim 4, wherein: The inputting the state information into the policy network, the policy network selecting the action information according to the current policy, outputting the optimized action information and calculating the signal-to-noise ratio and rate corresponding to the optimized action information, specifically includes: Comparing the state information with the target function; In response to the state information completely satisfying the objective function, the agent resource allocation is completed; In response to the state information being unable to fully satisfy the objective function, adjusting the value network and the policy network parameters based on the reward function, and updating the parameters of the resource allocation model; The optimized action information is used as action information, and the state information is used as state information, and is input into the resource allocation model again, and the action information is optimized until the state information output by the resource allocation model fully satisfies the objective function.

6. The multi-agent resource allocation method according to claim 5, wherein: The network environment model also includes a base station, an access point and a plurality of cellular terminals; wherein the cellular terminal is connected to the base station via an authorized frequency band for communication, and the power of the cellular terminal signal received by the base station is a first receiving power; in response to the action information and the optimized action information, part of the intelligent agents are connected to the authorized frequency band to communicate with the base station, and another part of the intelligent agents are connected to the unauthorized frequency band to communicate with the access point; the network environment model also includes environmental noise, and the power of the environmental noise is noise power; The state information includes: a first signal-to-noise ratio of the cellular terminal connected to the authorized frequency band, the agent's authorized and unauthorized frequency band selection, a second signal-to-noise ratio of the agent connected to the authorized frequency band, a first average rate of the agent connected to the unauthorized frequency band, and the number of users of the agent connected to the unauthorized frequency band; The adjusting the parameters of the network environment model based on the optimization action information to obtain the state information specifically includes: In response to the agent connecting to the authorized frequency band, the agent reuses the resource block position occupied by the cellular terminal, the power of the agent signal that is received by the base station and reuses the resource block is a first reuse power, and the sum of the powers of other agent signals that are received by the base station and reuse the resource block is a second received power; In response to the agent connecting to the unlicensed frequency band, a plurality of agents access the unlicensed frequency band by competing, and the power of the agent signal received by the access point is a third receiving power; Calculate the first signal-to-noise ratio based on the first received power, the first multiplexing power and the noise power; Calculate the second signal-to-noise ratio based on the first received power, the first multiplexing power, the second received power, and the noise power; Calculate a third signal-to-noise ratio of the agent connected to the unlicensed frequency band based on the third received power and the noise power, and calculate the first average rate based on the third signal-to-noise ratio; The state information is acquired based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the first average rate.

7. The multi-agent resource allocation method according to claim 6, wherein: The minimum signal-to-noise ratio for the cellular terminal to connect to the base station is set as a first signal-to-noise ratio threshold, the minimum signal-to-noise ratio for the agent to connect to the base station is set as a second signal-to-noise ratio threshold, and the minimum rate for the agent to connect to the access point is set as a first average rate threshold; The objective function is to maximize the total system throughput To optimize the target, solve the resource reuse scheme, where X represents the mode selection of the agent, F represents the channel selection of the agent, and P represents the power selection of the agent. is the throughput of the agent in the unlicensed frequency band, is the throughput of the intelligent agent in the licensed frequency band; The objective function further includes a first constraint function and a second constraint function; the first constraint function includes: the first signal-to-noise ratio is greater than the first signal-to-noise ratio threshold, and the second signal-to-noise ratio is greater than the second signal-to-noise ratio threshold; The second constraint function includes: the first average rate is greater than the first average rate threshold; The step of calculating the incentive value based on the signal-to-noise ratio and the rate, and inputting the action information into the value network so that the value network performs evaluation based on the current state information and the state information at the next moment to obtain the evaluation value, specifically includes: In response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive incentive value; In response to at least one of the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtaining a negative incentive value; A time difference error is calculated based on the evaluation value of the positive incentive value or the negative incentive value for the current state information, and parameters of the strategy network and the value network are optimized based on the time difference error, and the above steps are repeated.

8. A multi-agent resource allocation device, comprising: A modeling module, used to obtain action information of multiple agents, build a network environment model and a resource allocation model based on the multiple action information, and set an objective function; wherein the network environment model changes the network environment of the agents in response to the action information of the multiple agents, and each agent includes one resource allocation model; An optimization module, configured to obtain state information based on the network environment model, input the state information into the resource allocation model, and the resource allocation model optimizes the action information to obtain optimized action information; An adjustment module, used to adjust the parameters of the network environment model based on the optimization action information to obtain status information; An allocation module is used to allocate resources based on the state information and the objective function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to any one of claims 1 to 7. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images