Multi-agent resource allocation methods and related equipment

Abstract

This application provides a multi-agent resource allocation method, comprising: setting training rounds; firstly, initializing state information; setting incentive values ​​according to an optimized objective function; constructing a network environment model and a resource allocation model based on multiple action information; and obtaining initial action information of multiple agents. The network environment model is configured to respond to changes in the network environment in which the agents reside in response to multiple action information, and each agent includes a resource allocation model. State information is obtained based on the network environment model and input into the resource allocation model to optimize the action information and adjust the parameters of the network environment model to obtain optimized action information. State information is obtained based on the optimized action information. Resource allocation is performed based on the state information and the objective function. This method enables multiple agents to optimize resource allocation independently, improving the efficiency of the resource allocation algorithm.

Description

Technical Field

[0001] This application relates to the field of multi-agent deep reinforcement learning technology, and in particular to a multi-agent resource allocation method and related equipment. Background Technology

[0002] In recent years, with the increasing number of wireless terminal devices and the limited wireless spectrum resources, wireless spectrum resources are often reused in application scenarios where wireless devices are densely packed. However, due to mutual interference between wireless devices, it is necessary to allocate the reused wireless spectrum resources to ensure the communication quality of more devices.

[0003] For example, regarding the allocation of wireless resources in heterogeneous power network scenarios, licensed frequency band resources are becoming increasingly scarce. To ensure the communication needs of power services, it is necessary to consider the reuse of licensed and unlicensed spectrum resources by power multi-standard communication terminals. In actual typical power service scenarios such as substations and converter stations, WAPI (WLAN Authentication and Privacy Infrastructure) communication access has been implemented in power terminal communication. However, there is currently very little research on the reuse of licensed and unlicensed frequency bands by power terminals. Therefore, it is necessary to conduct research on wireless resource reuse in relevant scenarios.

[0004] Current optimization methods for spectrum resource allocation in wireless devices include traditional convex relaxation algorithms and machine learning algorithms. Compared to traditional algorithms, machine learning demonstrates significant advantages in solving complex problems. Machine learning algorithms use neural network models to perform deep learning optimization on the allocation results. However, when using traditional resource allocation methods to allocate resources for agents with multiple communication mode selections, resource selections, and power selections, the overall number of parameters is large, the algorithm complexity is high, and the convergence is slow, resulting in poor optimization efficiency. Therefore, a resource allocation method for agents that can reduce the number of parameters and improve optimization efficiency is needed. Summary of the Invention

[0005] In view of this, the purpose of this application is to propose a multi-agent resource allocation method and related equipment.

[0006] To achieve the above objectives, this application provides a multi-agent resource allocation method, comprising: setting training rounds; firstly, initializing state information; setting a reward function according to an optimized objective function; calculating incentive values ​​based on the reward function; constructing a network environment model and a resource allocation model based on the action information of multiple agents; and obtaining the action information of multiple agents in their initial states. The network environment model is configured to change the network environment in response to multiple action information, and each agent includes a resource allocation model. The resource allocation model includes a value network and a policy network. State information is obtained based on the network environment model and input into the resource allocation model. The policy network outputs optimized action information, the value network evaluates the state information and outputs an evaluation value, and a temporal difference error is calculated based on the evaluation value and the reward function. The parameters of the policy network and the value network are optimized based on the temporal difference error, resulting in an optimized policy, and the agent's action information is updated to the next state. The above iterative steps are repeated until the state information matches the objective function, and resource allocation is performed according to the optimized policy.

[0007] In some embodiments, the process involves: acquiring state information based on a network environment model; inputting the state information into a resource allocation model; the policy network outputting optimized action information; the value network evaluating the state information and outputting an evaluation value; and calculating a temporal difference error based on the evaluation value and the incentive value. Specifically, this includes: inputting state information into the policy network; the policy network selecting action information according to the current policy; outputting optimized action information and calculating the signal-to-noise ratio (SNR) and rate corresponding to the optimized action information; calculating incentive values ​​based on the SNR and rate; inputting the action information into the value network so that the value network evaluates the current state information and the state information at the next time step to obtain an evaluation value; calculating a temporal difference error based on the evaluation value and the incentive value; and updating the parameters of the policy network based on the temporal difference error.

[0008] In some embodiments, the method for calculating the incentive value specifically includes: the agent receives a positive incentive in response to the state information satisfying a first or second constraint function; the agent receives a negative incentive in response to the state information failing to satisfy the first or second constraint function. Based on the evaluation value and the incentive value, the weight value of the action information is adjusted through a policy network, specifically including: increasing the weight of the current action information in response to the state information approaching the objective function; and decreasing the weight of the current action information in response to the state information moving away from the objective function.

[0009] In some embodiments, the action information includes: the agent's operating mode, resource block location, and power level. The agent can switch between at least two operating modes, two resource block locations, and two power levels. The output layer of the policy network also includes a multi-head action network model, which comprises a mode action layer, a resource block action layer, and a power action layer. Updating the parameters of the policy network based on temporal differential error specifically includes: weighting the actions included in the action information based on their weight values, and calculating the probability distributions of the operating mode, resource block location, and power level respectively. Based on the weight values, the mode action layer calculates the mode probability distribution of the operating mode, the resource block action layer calculates the resource block probability distribution of the resource block location, and the power action layer calculates the power probability distribution of the power level. Based on the mode probability distribution, the mode action layer outputs the operating mode with the highest probability as the optimized operating mode. Based on the resource block probability distribution, the resource block action layer outputs the resource block location with the highest probability as the optimized resource block location. Based on the power probability distribution, the power action layer outputs the power level with the highest probability as the optimized power level. Based on optimized action patterns, optimized resource block locations, and optimized power levels, the multi-head action network model outputs optimized action information.

[0010] In some embodiments, the state information is input to the policy network, which selects action information according to the current policy, outputs optimized action information, and calculates the signal-to-noise ratio and rate corresponding to the optimized action information. Specifically, this includes comparing the state information with the objective function. If the state information fully satisfies the objective function, the agent's resource allocation is complete. If the state information does not fully satisfy the objective function, the parameters of the value network and policy network are adjusted based on the reward function, and the parameters of the resource allocation model are updated. The optimized action information is used as the action information, and the state information is used as the state information, and these are input again into the resource allocation model to optimize the action information until the state information output by the resource allocation model fully satisfies the objective function.

[0011] In some embodiments, the network environment model further includes a base station, an access point, and multiple cellular terminals. The cellular terminals communicate with the base station via licensed frequency bands, and the power of the cellular terminal signal received by the base station is the first received power. In response to action information and optimized action information, some agents connect to licensed frequency bands to communicate with the base station, while others connect to unlicensed frequency bands to communicate with the access point. The network environment model also includes ambient noise, the power of which is the noise power. State information includes: a first signal-to-noise ratio (SNR) for cellular terminals connecting to licensed frequency bands, agent selection of licensed and unlicensed frequency bands, a second SNR for agents connecting to licensed frequency bands, a first average rate for agents connecting to unlicensed frequency bands, and the number of users connecting to unlicensed frequency bands. Based on optimized action information, the parameters of the network environment model are adjusted to obtain state information, specifically including: in response to agents connecting to licensed frequency bands, the agent reuses the resource block location occupied by the cellular terminal; the power of the agent signal received by the base station from the reused resource block is the first reuse power; and the sum of the powers of other agent signals received by the base station from the reused resource block is the second received power. In response to an agent connecting to an unlicensed frequency band, multiple agents compete to access the unlicensed frequency band. The power of the agent signal received by the access point is the third received power. A first signal-to-noise ratio (SNR) is calculated based on the first received power, the first multiplexed power, and the noise power. A second SNR is calculated based on the first received power, the first multiplexed power, the second received power, and the noise power. A third SNR for the agent connecting to the unlicensed frequency band is calculated based on the third received power and the noise power, and a first average rate is calculated based on the third SNR. State information is obtained based on the first SNR, the second SNR, and the first average rate.

[0012] In some embodiments, the minimum signal-to-noise ratio (SNR) for a cellular terminal to connect to a base station is set as a first SNR threshold, the minimum SNR for an agent to connect to a base station is set as a second SNR threshold, and the minimum rate for an agent to connect to an access point is set as a first average rate threshold. The objective function aims to maximize the total system throughput. To optimize the objective, a resource reuse scheme is solved, where X represents the agent's mode selection, F represents the agent's channel selection, and P represents the agent's power selection. The throughput of the intelligent agent in the unlicensed frequency band. The objective function represents the throughput of the agent within the licensed frequency band. It also includes a first constraint function and a second constraint function. The first constraint function includes: a first signal-to-noise ratio (SNR) greater than a first SNR threshold, and a second SNR greater than a second SNR threshold. The second constraint function includes: a first average rate greater than a first average rate threshold. Incentive values ​​are calculated based on the SNR and rate. Action information is input into the value network so that the value network evaluates the current state information and the state information at the next moment, obtaining an evaluation value. Specifically, in response to the first SNR and the second SNR satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent receives a positive incentive value. In response to at least one of the first SNR and the second SNR failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent receives a negative incentive value. A temporal difference error is calculated based on the evaluation value of the current state information using the positive or negative incentive value. The parameters of the policy network and the value network are optimized based on the temporal difference error, and the above steps are repeated.

[0013] Based on the same inventive concept, this application also provides a multi-agent resource allocation device, comprising: a modeling module, configured to acquire action information of multiple agents, construct a network environment model and a resource allocation model based on the multiple action information, and set an objective function; wherein, the network environment model changes the network environment in response to the action information of the multiple agents, and each agent includes a resource allocation model; an optimization module, configured to acquire state information based on the network environment model, input the state information into the resource allocation model, and optimize the action information to acquire optimized action information; an adjustment module, configured to adjust the parameters of the network environment model based on the optimized action information to acquire state information; and an allocation module, configured to allocate resources based on the state information and the objective function.

[0014] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-agent resource allocation method as described in any of the above.

[0015] Based on the same inventive concept, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute any of the above-described multi-agent resource allocation methods.

[0016] As can be seen from the above, the multi-agent resource allocation method provided in this application enables multiple agents to optimize their own actions by setting a resource allocation model for each agent, thereby reducing the amount of optimization computation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this application;

[0019] Figure 2 A flowchart illustrating a multi-agent resource allocation method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a resource allocation model provided in an embodiment of this application;

[0021] Figure 4 A flowchart illustrating a method for obtaining optimized action information according to another embodiment of this application;

[0022] Figure 5 A flowchart illustrating a method for obtaining optimization action information based on weight values, provided in another embodiment of this application;

[0023] Figure 6 This is a schematic diagram illustrating the conversion of an output layer into a multi-head action network model, provided as an embodiment of this application.

[0024] Figure 7 A flowchart illustrating a method for determining whether resource allocation is complete, provided in another embodiment of this application;

[0025] Figure 8 A flowchart illustrating a method for obtaining status information according to another embodiment of this application;

[0026] Figure 9 A flowchart illustrating a method for determining resource allocation incentives for agents connected to different channels, provided in another embodiment of this application;

[0027] Figure 10 A schematic diagram of a resource allocation model provided in another embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of a multi-agent resource allocation device provided in an embodiment of this application;

[0029] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0031] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0032] In related technologies, neural network models are typically used to allocate spectrum resources for wireless devices. However, as the number of wireless devices increases, the number of parameters in traditional neural network models also increases. Due to limited computing resources, computing servers find it increasingly difficult to handle the ever-increasing number of model parameters. Therefore, it is necessary to design a wireless device spectrum resource allocation algorithm that can reduce algorithm complexity and decrease the computing power consumption during the optimization process.

[0033] In view of this, the present invention provides a multi-agent resource allocation method. By setting up a resource allocation model for multiple agents that need to allocate resources, each agent can determine whether its current action information satisfies the objective function based on its own observations, and optimize its own actions to ultimately obtain action information that meets the requirements of the objective function. By breaking down complex algorithms and distributing them to multiple agents to optimize their own action information, the number of model parameters is reduced.

[0034] refer to Figure 1This diagram illustrates an application scenario of the text correction method provided in this application. The application scenario includes a base station 101, an access point 201, multiple cellular terminals 102, and multiple intelligent agents 202. All cellular terminals 102 communicate with the base station 101 via licensed frequency band 301, and intelligent agents 202 can switch between licensed and unlicensed frequency bands 301 and 302. In response to the allocation result, some intelligent agents 202 communicate with the base station 101 via licensed frequency band 301, while others communicate with the access point 201 via unlicensed frequency band 302. At any given time, an intelligent agent 202 can only communicate via either licensed or unlicensed frequency band 302. Co-channel interference 303 exists between devices communicating via licensed frequency band 301.

[0035] The base station can provide cellular network signal access for smart agents and cellular terminals. The type of cellular network is not limited and can be selected reasonably according to the actual application needs. For example, it can be one or more combinations of GSM network, CDMA network, and 3G, 4G, and 5G network.

[0036] The multi-agent resource allocation method provided in this application embodiment can be applied to resource allocation scenarios of different types of intelligent agent systems. For example, intelligent agents can be power equipment in the power grid, urban road traffic equipment, urban public safety equipment, intelligent environmental monitoring equipment, etc.

[0037] The multi-agent resource allocation method of this application can be applied to scenarios such as power equipment, urban road traffic equipment, urban public safety equipment, and intelligent environmental monitoring equipment. The allocation target can be adjusted based on the signal strength requirements of agents in different application scenarios to adapt to the allocation of agent resources under different application scenarios.

[0038] The following is combined Figure 1 The application scenarios described above illustrate the multi-agent resource allocation method and related devices according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this regard. Rather, the embodiments of this application can be applied to any applicable scenario.

[0039] See Figure 2 This invention provides a multi-agent resource allocation method, comprising the following steps:

[0040] S1, Set up the training rounds. First, initialize the state information, set the incentive values ​​according to the optimized objective function, obtain the initial action information of multiple agents, and construct a network environment model and a resource allocation model based on the multiple action information. The network environment model is configured to respond to changes in the network environment in which the agent is located in response to multiple action information, and each agent includes a resource allocation model.

[0041] In this embodiment, the intelligent agent is an intelligent device with certain computing capabilities, capable of transmitting and receiving wireless signals. Based on adjustments to the intelligent agent's action information, the strength, frequency, and connected signal source of the signals emitted by the intelligent agent are altered. In such cases... Figure 1 The system shown includes multiple agents, each with a resource allocation model. This model adjusts the agent's actions based on the current state information fed back from the network environment model. Through multiple agents adjusting their actions, the system aims to optimize the allocation of wireless resources to all agents within the network environment model by ensuring that all agents satisfy the objective function.

[0042] The network environment model can be a mathematical model or a real network environment. When the network environment model is a mathematical model, the action information can be configured into the real network environment after the mathematical model is optimized, thereby reducing the impact on the network environment. When the network environment model is a real network environment, the optimization results of the resource allocation model can be directly applied to the real network environment, thereby improving the accuracy of the optimization results.

[0043] S2: Obtain state information based on the network environment model, input the state information into the resource allocation model, the policy network outputs actions, the value network evaluates the current state and outputs the evaluation value, and calculates the time difference error according to the evaluation value and the reward function.

[0044] In this context, the agent's state information changes along with its action information. The types of state information are not limited and can be selected appropriately based on the specific application requirements. For example, state information may include one or more of the following: interference experienced by the agent, the signal-to-noise ratio during signal transmission and reception, and the connection rate. The optimized action information includes the same types of information as the action information.

[0045] S3 optimizes the policy network parameters and value network parameters through temporal differential error, thereby optimizing the current policy and updating the agent's action to the next state.

[0046] Among them, the status information and the status information include the same types of information.

[0047] S4. Repeat the above loop steps until the state information matches the objective function, and then allocate resources according to the optimized strategy.

[0048] The repeated steps involve setting training rounds, setting incentive values ​​according to the optimized objective function, the agent interacting with the environment through the current policy, the agent selecting actions and obtaining incentive values ​​based on the current policy, the value network evaluating the action state and calculating the evaluation value, calculating the temporal difference error based on the evaluation value and the reward function, optimizing the policy network parameters and value network parameters through the temporal difference error, thus optimizing the current policy, updating the agent's action to the next state, and repeating the above steps until the training achieves the optimal resource allocation of the objective function.

[0049] In some embodiments, such as Figure 3 As shown, the resource allocation model also includes a value network and a policy network.

[0050] The resource allocation model optimizes the agent's action information based on the A2C (Actor-Critic) algorithm. Specifically, the Actor-Critic method consists of two artificial neural networks: the first network (Actor) approximates the policy π, and the second network (Critic) approximates the value function. The value network approximates the value of the state and the subsequent state, thereby guiding the actor's learning and further optimizing the policy network. Figure 2 The structure and general information flow of an Actor-Critic-based agent are shown, including the policy network receiving the state and outputting the probability distribution of actions in that state, and the value network receiving the state and reward for policy learning.

[0051] Specifically, the network environment model (environment) responds to the action information output by the policy network in the previous moment, outputting state information S. t With incentive information R t Value network acquires state information S t With incentive information R t Output evaluation information V(S) t The policy network responds to the state information S. t With evaluation information V(S) t Output optimized action information A t The network environment model (environment) responds to the optimization action information A output by the policy network. t Output status information S again t+1 With incentive information R t+1 The above process is repeated until the state information converges to the optimal state.

[0052] like Figure 4As shown, step S2 specifically includes:

[0053] S21, the state information is input into the policy network, the policy network selects an action according to the current policy probability, outputs the action and calculates the corresponding signal-to-noise ratio and rate.

[0054] S22, calculate the excitation value according to the above signal-to-noise ratio and rate, and input the above action into the value network so that the value network evaluates the action based on the current state and the next state information to obtain the evaluation value.

[0055] The value network is used to evaluate action information and output corresponding evaluation values. Action information is input into the value network and corresponding evaluation values ​​are obtained. The evaluation values ​​are output based on the current value network parameters.

[0056] S23. Based on the above evaluation values ​​and incentive values, the timing difference error is calculated using the TD error formula and used to update the parameters of the policy network mode layer, resource layer and power layer.

[0057] Specifically, the parameters of the value network and the policy network are adjusted based on the calculation of the temporal difference error, which is calculated by the following formula:

[0058]

[0059] in, Let γ represent the incentive value given by the reward function to agent k at state t+1, and let γ represent the discount factor, which takes a value between 0 and 1. (The discount factor parameter determines the importance of future rewards. A larger discount factor will make the agent pay more attention to future rewards, while a smaller discount factor will make the agent pay more attention to immediate rewards. It is generally taken as 0.95.) This represents the state information s of the value network for agent k. t The evaluation value, This represents the state information s of the value network for agent k. t+1 The evaluation value.

[0060] In some embodiments, the action information includes: the agent's operating mode, resource block location, and power level. The agent can switch between at least two operating modes, two resource block locations, and two power levels.

[0061] In this embodiment, the action information of the intelligent agent is represented by the following formula:

[0062]

[0063] Among them, X d Selecting X operating modes for agent d; RB dFor agent d, the location selection of resource block I is defined as 12 consecutive subcarriers in the frequency domain and 1 slot in the time domain, which is called 1 resource block RB. Since the bandwidth of 1 subcarrier is 15kHz, the bandwidth of 1 resource block is 180kHz. d,l For selecting L power levels for agent d, the power level of the agent is determined by its own factory parameters. The agent discretizes the power into L levels, which are determined by the following formula:

[0064]

[0065] The maximum power at which an intelligent agent transmits wireless signals.

[0066] The output layer of the policy network also includes a multi-head action network model, which includes a pattern action layer, a resource block action layer, and a power action layer.

[0067] Among them, such as Figure 6 As shown, the output layer of the policy network is replaced from the traditional output layer including action 1, action 2, action 3... action n, to a multi-head action network model including a pattern action layer, a resource block action layer, and a power action layer, and the output of the pattern action layer includes X. 1 X 2 …X n The action mode, resource block action layer output includes RB 1 RB 2 …RB F The resource block location, power action layer output includes P 1 P 2 …P L The power level is adjusted to reduce the number of parameters output by the policy network.

[0068] Specifically, in the output layer of a neural network model, the parameter with the highest probability is usually output in the form of a probability distribution. However, if the three parameters of the agent's action information are combined and output together, the number of units in the output layer becomes N×F×L. Taking N=2, F=50, and L=10 as an example, the number of units in the output layer is 1000. If the hidden layer before the output layer includes 200 units, then the weights in the last layer alone are equal to 200×1000=200000 (ignoring the bias term). However, by using a multi-head action network model, the weights of the three parameters in the action information can be calculated separately. The number of units in the output layer changes from N×F×L to N+F+L. When N=2, F=50, and L=10, the number of units in the output layer is 2+50+10=62. When the number of units in the hidden layer before the output layer is 200, the number of output weights is 200×62=12400, reducing the number of training parameters by about 16 times.

[0069] like Figure 5 As shown, step S3 specifically includes:

[0070] S31, based on the weight value of the action information, performs weighted calculation on the actions included in the action information, and calculates the probability distribution of working mode, resource block location and power level respectively.

[0071] S32, based on weight values, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block location, and the power action layer calculates the power probability distribution of the power level.

[0072] Among them, such as Figure 6 As shown, the pattern action layer, resource block action layer, and power action layer calculate the pattern probability distribution, resource block probability distribution, and power probability distribution, respectively. Based on the weight values, they increase the selection probability of the action mode, resource block position, and power level included in the action information corresponding to the state information that satisfies the objective function in the multi-head action network model, and decrease the selection probability of the action mode, resource block position, and power level included in the action information corresponding to the state information that does not satisfy the objective function in the multi-head action network model.

[0073] S33: Based on the mode probability distribution, the mode action layer outputs the operating mode with the highest probability, which is used as the optimized operating mode. Based on the resource block probability distribution, the resource block action layer outputs the resource block position with the highest probability, which is used as the optimized resource block position. Based on the power probability distribution, the power action layer outputs the power level with the highest probability, which is used as the optimized power level.

[0074] S34, based on optimized action patterns, optimized resource block locations, and optimized power levels, the multi-head action network model outputs optimized action information.

[0075] Specifically, the working mode, resource block location, and probability level with the highest probability are selected and output, and then combined as optimization action information.

[0076] The resource allocation method provided in this application sets up a policy network and a value network for each agent. The input layer of the value network includes state information. The value network calculates and outputs evaluation values ​​through hidden layers. The policy network takes the state information and evaluation values ​​as inputs and sets the output layer as a multi-head action network model to output the probability distribution of various action information.

[0077] In some embodiments, such as Figure 3 As shown, the resource allocation model also includes a reward function;

[0078] like Figure 7 As shown, step S4 specifically includes:

[0079] S41, compare the state information with the objective function.

[0080] The training process involves setting up training rounds, initializing state information, setting incentive values ​​according to the optimized objective function, interacting with the environment through the current policy, selecting actions based on policy probabilities and the incentive values ​​obtained, evaluating the action state through the value network and calculating the evaluation value, calculating the temporal difference error based on the evaluation value and the reward function, optimizing the policy network parameters and value network parameters through the temporal difference error, updating the agent's action to the next state, and repeating the above cyclical steps until training converges.

[0081] By comparing the state information with the objective function, it is determined whether the state information meets the optimization objective set in the objective function. The action information corresponding to the state information that meets the objective function and the action information corresponding to the state information that does not meet the objective function are processed separately.

[0082] S42, in response to the state information fully satisfying the objective function, the agent's resource allocation is completed.

[0083] In this process, resource allocation is completed when the state information fully satisfies the convergence state, according to the set training rounds. If the state information does not fully satisfy the training rounds or the convergence state, the parameters of the value network and policy network are adjusted based on the reward function, and the parameters of the resource allocation model are updated.

[0084] The reward function is calculated using the following formula:

[0085] When the agent accesses the authorized frequency band, the reward function is set to

[0086]

[0087] When an agent accesses an unlicensed frequency band, the reward function is set to...

[0088]

[0089] Specifically, when the state information satisfies the first constraint function, an excitation value is set, and the excitation value is... The excitation value is used to represent the agent's connection to the licensed frequency band. When the state information satisfies the second constraint function, the excitation value is set. The excitation value is used to represent the agent's connection to the unlicensed frequency band. When the state information cannot satisfy the first constraint function or the second constraint function, a penalty value is set, which is a constant -h. L and -h U Different penalty constants are set according to the different impacts on user capacity under authorized and unauthorized modes in order to accelerate convergence.

[0090] The parameters of the value network and policy network are adjusted based on the calculation of temporal difference error, which is calculated by the following formula:

[0091]

[0092] in, Let γ represent the incentive value given by the reward function to agent k at state t+1, and let γ represent the discount factor, which takes a value between 0 and 1. (The discount factor parameter determines the importance of future rewards. A larger discount factor will make the agent pay more attention to future rewards, while a smaller discount factor will make the agent pay more attention to immediate rewards. It is generally taken as 0.95.) This represents the state information s of the value network for agent k. t The evaluation value, This represents the state information s of the value network for agent k. t+1 The evaluation value.

[0093] The parameters of the value network are adjusted based on the following formula:

[0094]

[0095] in, This represents the value network parameters of agent k at the moment of obtaining state information. This represents the learning rate of the value network, which typically ranges from 0 to 1. A higher learning rate means that new information has a greater impact on the network and the update speed is faster, while a lower learning rate means that the update speed is slower and the network is more stable. It is generally set to around 0.001. This represents the policy gradient of the current value network; This indicates that the status information s t The evaluation value output by the value network after being input into agent k.

[0096] The parameters of the pattern selection layer in the policy network are adjusted based on the following formula:

[0097]

[0098] in, This represents the policy network parameters of the agent's k-mode selection layer at time t; This represents the action pattern output by the policy network at time t; s t This indicates that the above strategy approximates the output value of the neural network; α θ The network learning rate represents the policy; π θ Indicates the current strategy; This represents the policy gradient of the current policy network.

[0099] The parameters of the power selection layer in the policy network are adjusted based on the following formula:

[0100]

[0101] in, This represents the policy network parameters of the agent k power selection layer at time t; This represents the power level output by the policy network at time t.

[0102] The parameters of the resource selection layer in the policy network are adjusted based on the following formula:

[0103]

[0104] in, This represents the policy network parameters of agent k in the resource selection layer at time t; This represents the power level output by the policy network at time t.

[0105] S43, The optimized action information is used as action information, and the state information is used as state information. These are then input into the resource allocation model again to optimize the action information until the state information output by the resource allocation model fully satisfies the objective function.

[0106] Specifically, by using optimized action information as action information and state information as state information, and then re-inputting them into the resource allocation model, the action information is re-optimized. This optimization process is repeated until completion, thereby enabling the evaluation of different action information of the agent and ultimately selecting action information whose state information conforms to the objective function. Furthermore, by performing the same operation on multiple agents, the allocation of wireless resources to all agents is achieved.

[0107] In some embodiments, such as Figure 1 As shown, the network environment model also includes base stations, access points, and multiple cellular terminals. Cellular terminals communicate with base stations via licensed frequency bands, and the power of the signals received by the base stations from the cellular terminals is the first received power. In response to action information and optimized action information, some agents connect to licensed frequency bands to communicate with the base station, while others connect to unlicensed frequency bands to communicate with the access points. The network environment model also includes environmental noise, the power of which is the noise power. State information includes: the first signal-to-noise ratio (SNR) of cellular terminals connecting to licensed frequency bands, the second SNR of agents connecting to licensed frequency bands, and the first average rate of agents connecting to unlicensed frequency bands.

[0108] In this embodiment, the number of sub-channels in the licensed frequency band is denoted as N, expressed as N = {1,...,n}, where n is one sub-channel out of all channels; the total number of cellular terminals is denoted as C, expressed as C = {1,...,c}, where d is one of the cellular terminals; the total number of agents is denoted as D, expressed as D = {1,...,d}, where d is one of the agents; and the total bandwidth of the unlicensed frequency band is denoted as B. Environmental noise includes mutual interference between agents, between cellular devices, and between agents and cellular devices, as well as interference from other wireless signals in the area, and noise generated by signal attenuation, network congestion, equipment failure, etc.

[0109] like Figure 8 As shown, step S21 specifically includes:

[0110] S211, in response to the authorized frequency band for intelligent agent connection, the intelligent agent reuses the resource block location occupied by the cellular terminal, the power of the intelligent agent signal received by the base station from the reused resource block is the first reuse power, and the sum of the power of the intelligent agent signals received by the base station from other reused resource blocks is the second received power.

[0111] S212, in response to an agent connecting to an unlicensed frequency band, multiple agents compete to access the unlicensed frequency band, and the power of the agent signal received by the access point is the third received power.

[0112] S213, calculate the first signal-to-noise ratio based on the first received power, the first multiplexed power and the noise power.

[0113] The example provided is based on the scenario where all cellular terminals connect to the base station in advance via licensed frequency bands, and there is no co-channel interference between the cellular terminals:

[0114] The first signal-to-noise ratio can be calculated using the following formula:

[0115]

[0116] in, This represents the binary coefficient of the cellular terminal on channel resource n, when the cellular terminal occupies channel resource n. The value is 1 when the cellular terminal does not occupy channel resource n. The value is 0; This represents the binary coefficient of the agent on channel resource n, when the agent occupies channel resource n. The value is 1 when the agent has no channel resource n. The value of p is 0; c Indicates the wireless signal transmission power of a cellular terminal; g c,BS The channel gain of a cellular terminal is calculated using the following formula:

[0117] gc,BS =h c,BS β c,BS

[0118] h c,BS β represents the small-scale fading between the cellular terminal and the base station. c,BS This indicates large-scale fading between cellular terminals and base stations.

[0119] S214, calculate the second signal-to-noise ratio based on the first received power, the first multiplexed power, the second received power, and the noise power.

[0120] When agents reuse channel n, multiple agents may reuse a single channel. The second signal-to-noise ratio can be calculated using the following formula:

[0121]

[0122] Where d′ represents other agents in the multiplexing channel n. This represents the binary coefficient of the agent on channel resource n, when the agent occupies channel resource n. The value is 1 when the agent has no channel resource n. The value of p is 0; d Indicates the wireless signal transmission power of the intelligent agent; g d,BS The channel gain, representing the agent's connection to the licensed frequency band, is calculated using the following formula:

[0123] g d,BS =h d,BS β d,BS

[0124] h d,BS β represents the small-scale fading between the agent and the base station. d,BS N represents the large-scale fading between the agent and the base station; N0 represents the noise power.

[0125] S215, calculate the third signal-to-noise ratio of the agent connecting to the unlicensed frequency band based on the third received power and noise power, and calculate the first average rate based on the third signal-to-noise ratio.

[0126] The third signal-to-noise ratio is calculated using the following formula:

[0127]

[0128] in, This indicates the power of the intelligent agent connecting to unlicensed frequency bands. The channel gain for an agent connecting to unlicensed frequency bands is calculated using the following formula:

[0129]

[0130] This represents the small-scale fading between the agent and the access point. This indicates large-scale fading between the agent and the access point.

[0131] The first average rate is calculated based on the third signal-to-noise ratio, using the following formula:

[0132]

[0133] Among them, B u This indicates the total bandwidth of the unlicensed frequency band. This represents the binary coefficients of the agent in the unlicensed frequency band, when the agent occupies the unlicensed frequency band. The value is 1 when the agent is not occupying the unlicensed frequency band. The value is 0.

[0134] S216, obtain state information based on the first signal-to-noise ratio, the second signal-to-noise ratio and the first average rate.

[0135] In some embodiments, the constraint function includes a first constraint function and a second constraint function;

[0136] like Figure 9 As shown, step S22 specifically includes:

[0137] S221, in response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive excitation value.

[0138] S222, in response to at least one of the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtains a negative excitation value.

[0139] The first constraint function is used to determine whether an agent connecting to a base station via a licensed frequency band meets the resource allocation requirements, and the second constraint function is used to determine whether an agent connecting to an access point via an unlicensed frequency band meets the resource allocation requirements.

[0140] S223, in response to the above incentive values, the value network calculates the evaluation values ​​of the previous state and the next state respectively, and calculates the temporal difference error according to the TD error formula, which is used to optimize the parameters of the policy network and the value network.

[0141] All the above state information is input into the resource allocation model as state information, and the optimization action information is used as action information, and the action information is optimized again.

[0142] Specifically, by using optimized action information as action information and state information as state information, and then re-inputting them into the resource allocation model, the action information is re-optimized. This optimization process is repeated until completion, thereby enabling the evaluation of different action information of the agent and ultimately selecting action information whose state information conforms to the objective function. Furthermore, by performing the above operation on each agent, the allocation of wireless resources for all agents is achieved.

[0143] In some embodiments, the minimum signal-to-noise ratio (SNR) for a cellular terminal to connect to a base station is set as a first SNR threshold, the minimum SNR for an intelligent agent to connect to a base station is set as a second SNR threshold, and the minimum rate for an intelligent agent to connect to an access point is set as a first average rate threshold. The first constraint function includes: a first SNR greater than the first SNR threshold, and a second SNR greater than the second SNR threshold. The second constraint function includes: a first average rate greater than the first average rate threshold.

[0144] In this embodiment, the goal is to maximize the total system throughput. Let X be the objective function, and F be the channel selection, and P be the power selection. For end users' throughput in unlicensed frequency bands, This refers to the throughput of end users within the licensed frequency band.

[0145] When solving the above objective function, the following constraints also need to be satisfied:

[0146] The first constraint function includes:

[0147]

[0148] The second constraint function includes:

[0149]

[0150] The objective function may also include a first constraint, a second constraint, and a third constraint.

[0151] The first constraint includes:

[0152]

[0153] The second constraint includes:

[0154]

[0155] The third constraint includes:

[0156] 0≤p d ≤P max

[0157] Where, p d P represents the transmission power of the agent's wireless signal. max This indicates the maximum power at which the agent transmits wireless signals, and is determined by the agent's own performance parameters. This represents the first signal-to-noise ratio threshold. This represents the second signal-to-noise ratio threshold, which can be reasonably configured according to resource allocation requirements. The first constraint indicates that the terminal's licensed and unlicensed channel mode selection has a binary value; the second constraint is that the terminal can only select one communication mode; and the third constraint is the terminal's maximum power limit.

[0158] As a preferred implementation method, such as Figure 10 As shown, the multi-agent resource allocation method provided in this application also includes a resource allocation network model as shown in the figure. The policy network (Actor network) outputs actions to change environmental parameters. The environment outputs state information to the value network (Critic network). The value network includes an input layer, three hidden layers (the number of hidden layers needs to take into account multiple factors such as complexity and actual problems, and appropriate parameters can be selected) and an output layer. The state information is allocated to the three hidden layers through the input layer of the value network. The hidden layers perform weighted calculations on the state information and output the state value as the advantage function through the output layer. The state information of the advantage function and the environment output is input into the policy network. The input layer of the policy network distributes the advantage function and state information to the three hidden layers of the policy network. The three hidden layers perform weighted calculations on the state information and the advantage function, and classify the corresponding action information through the softmax activation function. The action information of the agent is split and output through the pattern action layer, resource block (RB) action layer and power action layer in the multi-head action network model (hierarchical multi-head policy), respectively. The probability distributions of multiple pattern actions, multiple resource block actions and multiple power actions are given. The pattern action, resource block action and power action with the highest probability are used as the optimized action information. The environment is adjusted based on the actions in the optimized action information.

[0159] As can be seen from the above embodiments of this application, this application sets a resource allocation model for each agent, enabling multiple agents to optimize their own actions, thereby reducing the computational load of optimization. By configuring the resource allocation model as an A2C network model including a value network and a policy network, the value function compares the state information with the objective function, increasing the weight of action information that satisfies the objective function and decreasing the weight of action information that does not satisfy the objective function, thus enabling the allocation structure of the resource allocation model to converge. Furthermore, setting the output layer of the policy network as a multi-head structure reduces the number of weights that need to be calculated during the optimization process, speeding up the convergence speed and improving the utilization rate of wireless resources.

[0160] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0161] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0162] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a multi-agent resource allocation device.

[0163] refer to Figure 11 The multi-agent resource allocation device includes:

[0164] Modeling module 501 is used to acquire action information from multiple agents, construct a network environment model and a resource allocation model based on the action information, and set an objective function. The network environment model responds to the action information of the multiple agents by changing the network environment in which the agents are located, and each agent includes a resource allocation model.

[0165] The optimization module 502 is used to obtain state information based on the network environment model, input the state information into the resource allocation model, optimize the action information, and obtain optimized action information.

[0166] The adjustment module 503 is used to adjust the parameters of the network environment model based on the optimization action information and obtain state information.

[0167] The allocation module 504 is used to allocate resources based on state information and objective function.

[0168] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0169] The apparatus described above is used to implement the corresponding multi-agent resource allocation method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0170] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-agent resource allocation method described in any of the above embodiments.

[0171] Figure 12 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0172] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0173] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0174] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0175] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0176] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0177] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0178] The electronic devices described above are used to implement the corresponding multi-agent resource allocation methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0179] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the multi-agent resource allocation method as described in any of the above embodiments.

[0180] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0181] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the multi-agent resource allocation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0182] It should be noted that the embodiments of this application can also be further described in the following ways:

[0183] A multi-agent resource allocation method, comprising:

[0184] The training rounds are set up by first initializing the state information, then setting the reward function according to the optimized objective function, calculating the incentive value based on the reward function, and constructing a network environment model and a resource allocation model based on the action information of multiple agents. The action information of multiple agents in their initial states is obtained. The network environment model is configured to change the network environment in response to multiple action information, and each agent includes a resource allocation model. The resource allocation model includes a value network and a policy network.

[0185] The state information is obtained based on the network environment model. The state information is input into the resource allocation model. The policy network outputs optimization action information. The value network evaluates the state information and outputs the evaluation value. The time difference error is calculated based on the evaluation value and the reward function.

[0186] The parameters of the policy network and the value network are optimized based on the temporal difference error, so that the current policy is optimized and the agent's action information is updated to the next state. The above loop steps are repeated until the state information is consistent with the objective function, and resources are allocated according to the optimized policy.

[0187] Optionally, the process involves obtaining state information based on a network environment model, inputting this state information into a resource allocation model, the policy network outputting optimization action information, the value network evaluating the state information and outputting an evaluation value, and calculating the time-series difference error based on the evaluation value and the incentive value. Specifically, this includes:

[0188] The state information is input into the policy network, which selects action information according to the current policy, outputs optimized action information, and calculates the signal-to-noise ratio and rate corresponding to the optimized action information.

[0189] The excitation value is calculated based on the signal-to-noise ratio and the rate. The action information is then input into the value network so that the value network can evaluate based on the current state information and the state information at the next moment, and obtain the evaluation value.

[0190] The temporal difference error is calculated based on the evaluation value and the incentive value, and the parameters of the policy network are updated based on the temporal difference error.

[0191] Optional, of which,

[0192] The specific methods for calculating the excitation value include:

[0193] The agent receives a positive incentive in response to the state information satisfying the first or second constraint function.

[0194] In response to the state information failing to satisfy the first or second constraint function, the agent receives a negative incentive.

[0195] Based on the evaluation and incentive values, the weights of the action information are adjusted through a policy network, specifically including:

[0196] In response to the state information approaching the objective function, the weight of the current action information is increased.

[0197] In response to the state information moving away from the objective function, the weight of the current action information is reduced.

[0198] Optional, of which,

[0199] Action information includes: the agent's operating mode, resource block location, and power level. Specifically, the agent must be able to switch between at least two operating modes, two resource block locations, and two power levels.

[0200] The output layer of the policy network also includes a multi-head action network model, which includes a pattern action layer, a resource block action layer, and a power action layer.

[0201] The parameters of the network based on the temporal difference error update strategy include:

[0202] Based on the weight values ​​of the action information, the actions included in the action information are weighted and calculated to calculate the probability distribution of working mode, resource block location and power level respectively.

[0203] Based on the weight values, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block location, and the power action layer calculates the power probability distribution of the power level.

[0204] Based on the mode probability distribution, the mode action layer outputs the operating mode with the highest probability, which is used as the optimized operating mode. Based on the resource block probability distribution, the resource block action layer outputs the resource block location with the highest probability, which is used as the optimized resource block location. Based on the power probability distribution, the power action layer outputs the power level with the highest probability, which is used as the optimized power level.

[0205] Based on optimized action patterns, optimized resource block locations, and optimized power levels, the multi-head action network model outputs optimized action information.

[0206] Optional, of which,

[0207] The state information is input into the policy network. The policy network selects action information according to the current policy, outputs optimized action information, and calculates the signal-to-noise ratio and rate corresponding to the optimized action information. Specifically, this includes:

[0208] The state information is compared with the objective function.

[0209] If the state information fully satisfies the objective function, then the agent's resource allocation is complete.

[0210] If the state information cannot fully satisfy the objective function, the parameters of the value network and policy network are adjusted based on the reward function, and the parameters of the resource allocation model are updated.

[0211] The optimized action information is used as action information, and the state information is used as state information. These are then input into the resource allocation model again to optimize the action information until the state information output by the resource allocation model fully satisfies the objective function.

[0212] Optional, of which,

[0213] The network environment model also includes base stations, access points, and multiple cellular terminals. Cellular terminals communicate with base stations via licensed frequency bands, and the power of the signals received by the base stations from the cellular terminals is the first received power. In response to action information and optimized action information, some agents connect to licensed frequency bands to communicate with the base station, while others connect to unlicensed frequency bands to communicate with the access points. The network environment model also includes ambient noise, the power of which is the noise power.

[0214] The status information includes: the first signal-to-noise ratio of the cellular terminal connecting to the licensed frequency band, the agent's selection of licensed and unlicensed frequency bands, the second signal-to-noise ratio of the agent connecting to the licensed frequency band, the first average rate of the agent connecting to the unlicensed frequency band, and the number of users of the agent connecting to the unlicensed frequency band.

[0215] Based on the optimized action information, the parameters of the network environment model are adjusted to obtain state information, specifically including:

[0216] In response to the authorized frequency band for intelligent agent connection, the intelligent agent reuses the resource block location occupied by the cellular terminal. The power of the intelligent agent signal received by the base station from the reused resource block is the first reuse power, and the sum of the power of other intelligent agent signals received by the base station from the reused resource block is the second received power.

[0217] In response to an agent connecting to an unlicensed frequency band, multiple agents compete to access the unlicensed frequency band, and the power of the agent signal received by the access point is the third received power.

[0218] The first signal-to-noise ratio is calculated based on the first received power, the first multiplexed power, and the noise power.

[0219] The second signal-to-noise ratio is calculated based on the first received power, the first multiplexed power, the second received power, and the noise power.

[0220] The third signal-to-noise ratio (SNR) of the agent connected to the unlicensed frequency band is calculated based on the third received power and noise power, and the first average rate is calculated based on the third SNR.

[0221] State information is obtained based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the first average rate.

[0222] Optional, of which,

[0223] The minimum signal-to-noise ratio (SNR) for a cellular terminal to connect to a base station is set as a first SNR threshold, the minimum SNR for an intelligent agent to connect to a base station is set as a second SNR threshold, and the minimum rate for an intelligent agent to connect to an access point is set as a first average rate threshold.

[0224] Objective function to maximize total system throughput To optimize the objective, a resource reuse scheme is solved, where X represents the agent's mode selection, F represents the agent's channel selection, and P represents the agent's power selection. The throughput of the intelligent agent in the unlicensed frequency band. This represents the throughput of the intelligent agent within the licensed frequency band.

[0225] The objective function also includes a first constraint function and a second constraint function. The first constraint function includes: a first signal-to-noise ratio greater than a first signal-to-noise ratio threshold, and a second signal-to-noise ratio greater than a second signal-to-noise ratio threshold.

[0226] The second constraint function includes: the first average rate is greater than the first average rate threshold.

[0227] The excitation value is calculated based on the signal-to-noise ratio and rate. The action information is then input into the value network, enabling the value network to evaluate based on the current state information and the state information at the next moment, thereby obtaining an evaluation value. Specifically, this includes:

[0228] In response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive excitation value.

[0229] In response to at least one of the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtains a negative excitation value.

[0230] Calculate the temporal difference error based on the evaluation value of the current state information based on the positive or negative incentive value, optimize the parameters of the policy network and the value network based on the temporal difference error, and repeat the above steps.

[0231] A multi-agent resource allocation device, comprising:

[0232] The modeling module is used to acquire action information from multiple agents, construct a network environment model and a resource allocation model based on this action information, and set the objective function. Specifically, the network environment model responds to the action information of the multiple agents by changing the network environment in which the agents are located, and each agent includes a resource allocation model.

[0233] The optimization module is used to obtain state information based on the network environment model, input the state information into the resource allocation model, and optimize the action information to obtain optimized action information.

[0234] The adjustment module is used to adjust the parameters of the network environment model based on the optimization action information and obtain state information.

[0235] The allocation module is used to allocate resources based on state information and objective functions.

[0236] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any of the above.

[0237] A non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform any of the methods described above.

[0238] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0239] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0240] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0241] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A multi-agent resource allocation method, comprising: setting a training round, first initializing state information, setting a reward function according to an optimized objective function, calculating an incentive value based on the reward function, constructing a network environment model and a resource allocation model based on action information of a plurality of agents, and obtaining the action information of the plurality of agents in the initial state; wherein the network environment model is configured to change the network environment in which the agents are located in response to a plurality of action information, each of the agents comprises a resource allocation model; the resource allocation model comprises a value network and a policy network; the action information comprises: the working mode, resource block position and power level of the agent; the output layer of the policy network further comprises a multi-head action network model, the multi-head action network model comprises a mode action layer, a resource block action layer and a power action layer; based on the network environment model, the state information is obtained, the state information is input into the resource allocation model, the policy network outputs the optimized action information, the value network evaluates the state information and outputs the evaluation value, the time difference error is calculated based on the evaluation value and the reward function, specifically including: inputting the state information into the policy network, the policy network selecting the action information according to the current policy, outputting the optimized action information and calculating the signal-to-noise ratio and rate corresponding to the optimized action information; based on the signal-to-noise ratio and the rate, the incentive value is calculated, and the action information is input into the value network, so that the value network evaluates based on the current state information and the state information at the next time to obtain the evaluation value; based on the evaluation value and the incentive value, the time difference error is calculated, and the parameters of the policy network are updated based on the time difference error, specifically including: based on the weight value of the action information, the actions included in the action information are weighted and calculated, and the probability distribution of the working mode, the resource block position and the power level is calculated respectively; based on the weight value, the mode action layer calculates the mode probability distribution of the working mode, the resource block action layer calculates the resource block probability distribution of the resource block position, and the power action layer calculates the power probability distribution of the power level; based on the mode probability distribution, the mode action layer outputs the working mode with the highest probability as the optimized working mode; based on the resource block probability distribution, the resource block action layer outputs the resource block position with the highest probability as the optimized resource block position; based on the power probability distribution, the power action layer outputs the power level with the highest probability as the optimized power level; based on the optimized working mode, the optimized resource block position and the optimized power level, the multi-head action network model outputs the optimized action information; based on the time difference error, the parameters of the policy network and the parameters of the value network are optimized, so that the current policy is optimized, and the action information of the agent is updated to the next state; The above loop steps are repeated until the state information is consistent with the target function, and resource allocation is performed according to the optimized strategy.

2. The multi-agent resource allocation method of claim 1, wherein, The calculation method of the incentive value specifically comprises: In response to the state information satisfying the first or second constraint function, the agent obtains a positive incentive; In response to the state information not satisfying the first or second constraint function, the agent obtains a negative incentive; Based on the evaluation value and the incentive value, the weight value of the action information is adjusted through the policy network, specifically comprising: In response to the state information being close to the target function, the weight of the current action information is increased; In response to the state information being far away from the target function, the weight of the current action information is reduced.

3. The multi-agent resource allocation method of claim 2, wherein, wherein The agent can switch between at least 2 working modes, 2 resource block positions, and 2 power levels.

4. The multi-agent resource allocation method of claim 3, wherein, The state information is input into the policy network, the policy network selects the action information according to the current strategy, outputs the optimized action information, and calculates the signal-to-noise ratio and the rate corresponding to the optimized action information, specifically comprising: The state information is compared with the target function; In response to the state information completely satisfying the target function, the agent resource allocation is completed; In response to the state information not completely satisfying the target function, the value network and the policy network parameters are adjusted based on the reward function, and the parameters of the resource allocation model are updated; The optimized action information is input as action information, and the state information is input as state information again to the resource allocation model to optimize the action information until the state information output by the resource allocation model completely satisfies the target function.

5. The multi-agent resource allocation method of claim 4, wherein, The network environment model further comprises a base station, an access point, and a plurality of cellular terminals; wherein the cellular terminals communicate with the base station through a licensed frequency band, and the power of the signal received by the base station from the cellular terminals is a first received power; in response to the action information and the optimized action information, part of the agents communicate with the base station through the licensed frequency band, and another part of the agents communicate with the access point through an unlicensed frequency band; the network environment model further comprises environmental noise, and the power of the environmental noise is a noise power; The state information comprises: a first signal-to-noise ratio of the cellular terminal connecting the licensed frequency band, an authorized and unlicensed frequency band selection of the agent, a second signal-to-noise ratio of the agent connecting the licensed frequency band, a first average rate of the agent connecting the unlicensed frequency band, and a number of users of the agent connecting the unlicensed frequency band; Based on the optimized action information, the parameters of the network environment model are adjusted to obtain state information, specifically comprising: In response to the agent connecting to the licensed frequency band, the agent reuses the resource block location occupied by the cellular terminal. The power of the agent signal multiplexed by the base station is the first reuse power, and the sum of the power of other agent signals multiplexed by the base station is the second receive power. In response to the intelligent agent connecting to the unlicensed frequency band, multiple intelligent agents compete to access the unlicensed frequency band, and the power of the signal received by the access point from the intelligent agent is the third received power; The first signal-to-noise ratio is calculated based on the first received power, the first multiplexed power, and the noise power. The second signal-to-noise ratio is calculated based on the first received power, the first multiplexed power, the second received power, and the noise power. The third signal-to-noise ratio of the agent connecting to the unlicensed frequency band is calculated based on the third received power and the noise power, and the first average rate is calculated based on the third signal-to-noise ratio; The state information is obtained based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the first average rate.

6. The multi-agent resource allocation method according to claim 5, wherein, The minimum signal-to-noise ratio (SNR) for the cellular terminal to connect to the base station is set as a first SNR threshold, the minimum SNR for the intelligent agent to connect to the base station is set as a second SNR threshold, and the minimum rate for the intelligent agent to connect to the access point is set as a first average rate threshold. The objective function is to maximize the total throughput of the system Solving the resource reuse scheme as an optimization target, wherein X represents the mode selection of the agent, F represents the channel selection of the agent, P represents the power selection of the agent, The throughput of the agent in the unlicensed frequency band, The throughput of the agent in the licensed frequency band, d represents an agent, and D represents a set of agents. The objective function further includes a first constraint function and a second constraint function; the first constraint function includes: the first signal-to-noise ratio is greater than the first signal-to-noise ratio threshold, and the second signal-to-noise ratio is greater than the second signal-to-noise ratio threshold. The second constraint function includes: the first average rate is greater than the first average rate threshold; The process of calculating the excitation value based on the signal-to-noise ratio and the rate, and inputting the action information into the value network so that the value network can evaluate based on the current state information and the state information at the next moment to obtain the evaluation value, specifically includes: In response to the first signal-to-noise ratio and the second signal-to-noise ratio satisfying the first constraint function, or the first average rate satisfying the second constraint function, the agent obtains a positive excitation value; In response to the first signal-to-noise ratio and the second signal-to-noise ratio failing to satisfy the first constraint function, or the first average rate failing to satisfy the second constraint function, the agent obtains a negative excitation value; Calculate the temporal difference error based on the evaluation value of the current state information using the positive or negative incentive value, optimize the parameters of the policy network and the value network based on the temporal difference error, and repeat the above steps.

7. A multi-agent resource allocation apparatus for performing the method of any one of claims 1-6, comprising: A modeling module configured to obtain action information of a plurality of agents, construct a network environment model and a resource allocation model based on the action information, and set an objective function, wherein the network environment model changes a network environment in which the agents exist in response to the action information of the agents, and each of the agents includes one of the resource allocation models; An optimization module configured to obtain state information based on the network environment model, input the state information into the resource allocation model, optimize the action information by using the resource allocation model, and obtain optimized action information; An adjustment module configured to adjust parameters of the network environment model based on the optimized action information, and obtain state information; A distribution module configured to distribute resources based on the state information and the objective function. 8.An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any one of claims 1 to 6. 9.A non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the method of any one of claims 1 to 6.

Images