Communication Resource Allocation Method, Device, Electronic Device and Medium of a Communication System

Abstract

The present application relates to the field of mobile communications technology, and more particularly to a method, apparatus, electronic device, and medium for allocating communication resources in a communication system. The method comprises: obtaining environmental parameter information for each time slot in the communication system; identifying environmental parameters for a primary user and a secondary user in the environmental parameter information; inputting the environmental parameters for the primary user and the secondary user into a pre-trained allocation model, outputting communication resource allocation strategies for the primary user and the secondary user, and utilizing the communication resources to allocate the communication resources of the communication system. This solves the problem that the dynamic intelligent allocation of multiple communication resources in a related mobile communication system fails to maximize user benefits and suffers from low communication system performance.

Description

Technical Field

[0001] The present application relates to the field of mobile communication technology, and in particular to a communication resource allocation method, device, electronic device and medium for a communication system. Background Art

[0002] In recent years, with the continuous development of mobile communication technology, the scale of data traffic has rapidly increased. However, existing licensed frequency bands are becoming increasingly saturated and underutilized. At the same time, the emergence and implementation of new technologies such as the massive Internet of Things and the Internet of Vehicles (IoV) are placing higher demands on wireless communication services, making it difficult for existing spectrum resources to support these demands. To alleviate this conflict and better utilize spectrum resources, researchers have proposed using techniques and algorithms such as game theory and deep reinforcement learning to allocate communication resources in unlicensed frequency bands, thereby improving spectrum utilization.

[0003] Game theory, a mathematical theory and method, is a branch of modern mathematics that studies the optimization strategies used by individuals to maximize their benefits in competitive or combative games. Deep reinforcement learning combines the advantages of both deep and reinforcement learning technologies, providing users with the intelligence to solve complex decision-making problems. In cognitive radio systems, users need to develop access strategies that maximize their benefits while minimizing interference with existing systems. Some existing technologies have considered applying game theory or deep reinforcement learning to the allocation of communication resources in mobile communication systems. However, the question of how to combine game theory and deep reinforcement learning to leverage their strengths for dynamic and intelligent resource allocation and improve communication performance remains to be answered. Summary of the Invention

[0004] The present application provides a communication resource allocation method, device, electronic device and medium for a communication system to solve the problems that the dynamic intelligent allocation of multiple communication resources in the related art mobile communication system cannot maximize the user's benefits and has low communication system performance.

[0005] The first aspect of the present application provides a communication resource allocation method for a communication system, comprising the following steps: obtaining environmental parameter information of each time slot in the communication system; identifying the environmental parameters of a primary user and the environmental parameters of a secondary user in the environmental parameter information; inputting the environmental parameters of the primary user and the environmental parameters of the secondary user into a pre-trained allocation model, outputting a communication resource allocation strategy for the primary user and the secondary user, and using the communication resources to allocate the communication resources of the communication system.

[0006] Optionally, in one embodiment of the present application, the allocation model is obtained by training based on the environmental parameter training data of the primary user and the secondary user, including: constructing an artificial neural network structure according to the system model and optimization objectives of the communication system; using the environmental parameters of the primary user as the input of the artificial neural network corresponding to the primary user, and using the environmental parameters of the secondary user as the input of the artificial neural network corresponding to the secondary user; training the artificial neural network structure through a preset deep reinforcement learning algorithm to obtain the allocation model.

[0007] Optionally, in one embodiment of the present application, the preset deep reinforcement learning algorithm includes a deep Q network and a deep recursive Q network, and the artificial neural network structure is trained by the preset deep reinforcement learning algorithm to obtain the allocation model, including: using the deep Q network and the deep recursive Q network to generate a DQN network for each user according to the user's demand strategy; in the process of data generation, for each DQN network, the system state is input into the DQN network in each time slot, and the sequence number of the largest item in the network output is used as the alternative action, and a decision is made according to a preset method, wherein the intelligent agent selects the alternative action with a probability of 1-ε and randomly selects an action with a probability of ε; in the process of updating the network parameters, the data generated by the training process is input into the DQN network, and the new network parameters are calculated according to the network update algorithm until the preset conditions are met to obtain the allocation model.

[0008] Optionally, in one embodiment of the present application, a frozen target network technology or an experience replay technology is used for training during the training process, wherein, if the DQN algorithm using the frozen target network technology is used, the target network and the prediction network of the DQN network are used to respectively calculate the true value and the estimated value of the Q value, and the training loss is calculated based on the true value and the estimated value. When the training loss is less than the preset loss, the iterative training is stopped; if the DQN algorithm using the experience replay technology is used, the data generated during training is stored in a preset memory. When updating the neural network parameters, the intelligent agent randomly extracts a portion of the data from the preset memory and scrambles it, and applies the network update algorithm for training.

[0009] Optionally, in one embodiment of the present application, before model training, it also includes: obtaining the system composition, user information, resource type to be allocated and optimization goal of the communication system; matching the game theory model according to the system composition, the user information, the resource type and the optimization goal, and modeling the communication system resource allocation problem according to the game theory model, and designing the utility function of each user; according to the established game theory model and the user utility function, the leader and follower in the game theory model are defined as two intelligent agents, the interference price set by the main user for each secondary user is defined as the action of the main user, and the channel to be accessed by the secondary user and the selected access power are defined as the action of the secondary user; the channel and power accessed by each secondary user in the previous time slot are defined as the environmental parameters of the main user, and the interference price set by each main user for a secondary user is defined as the environmental parameters of the secondary user.

[0010] Optionally, in one embodiment of the present application, the utility function of the primary user is designed based on the primary user's income and consumed power; the utility function of the secondary user is designed based on the price paid by the secondary user and the quality of communication; and the utility function of the primary user in the game theory model is defined as the reward function of the intelligent agent corresponding to the primary user, and the utility function of the secondary user is defined as the reward function of the intelligent agent corresponding to the secondary user.

[0011] The second aspect embodiment of the present application provides a communication resource allocation device for a communication system, including: a first acquisition module, used to obtain environmental parameter information of each time slot in the communication system; an identification module, used to identify the environmental parameters of the primary user and the environmental parameters of the secondary user in the environmental parameter information; an allocation module, used to input the environmental parameters of the primary user and the environmental parameters of the secondary user into a pre-trained allocation model respectively, output the communication resource allocation strategy of the primary user and the secondary user, and use the communication resources to allocate the communication resources of the communication system.

[0012] Optionally, in one embodiment of the present application, the allocation model is trained based on environmental parameter training data of the primary user and the secondary user.

[0013] Optionally, in one embodiment of the present application, it also includes: a training module, used to construct an artificial neural network structure based on the system model and optimization objectives of the communication system; using the environmental parameters of the primary user as the input of the artificial neural network corresponding to the primary user, and using the environmental parameters of the secondary user as the input of the artificial neural network corresponding to the secondary user; training the artificial neural network structure through a preset deep reinforcement learning algorithm to obtain the allocation model.

[0014] Optionally, in one embodiment of the present application, the preset deep reinforcement learning algorithm includes a deep Q network and a deep recursive Q network, and the training module is further used to use the deep Q network and the deep recursive Q network to generate a DQN network for each user according to the user's demand strategy; in the process of data generation, for each DQN network, the system state is input into the DQN network in each time slot, and the sequence number of the largest item in the network output is used as the alternative action, and a decision is made according to a preset method, wherein the intelligent agent selects the alternative action with a probability of 1-ε and randomly selects an action with a probability of ε; in the process of updating the network parameters, the data generated by the training process is input into the DQN network, and the new network parameters are calculated according to the network update algorithm until the preset conditions are met to obtain the allocation model.

[0015] Optionally, in one embodiment of the present application, the training module is further used to perform training using a frozen target network technology or an experience replay technology during the training process, wherein, if the DQN algorithm using the frozen target network technology is used, the target network and the prediction network of the DQN network are used to respectively calculate the true value and the estimated value of the Q value, and the training loss is calculated based on the true value and the estimated value, until the training loss is less than the preset loss, the iterative training is stopped; if the DQN algorithm using the experience replay technology is used, the data generated during training is stored in a preset memory, and when updating the neural network parameters, the intelligent agent randomly extracts a portion of the data from the preset memory and scrambles it, and applies the network update algorithm for training.

[0016] Optionally, in one embodiment of the present application, it also includes: a second acquisition module, used to obtain the system composition, user information, resource type to be allocated and system optimization objectives of the communication system before model training; a processing module, used to match the game theory model according to the system composition, the user information, the resource type and the optimization objective, and model the communication system resource allocation problem according to the game theory model, and design the utility function of each user; a definition module, used to define the leader and follower in the game theory model as two intelligent agents according to the established game theory model and the user utility function, define the interference price set by the main user for each secondary user as the action of the main user, define the channel to be accessed by the secondary user and the selected access power as the action of the secondary user; define the channel and power accessed by each secondary user in the previous time slot as the environmental parameters of the main user, and define the interference price set by each main user for a certain secondary user as the environmental parameters of the secondary user.

[0017] Optionally, in one embodiment of the present application, the utility function of the primary user is designed based on the primary user's income and consumed power; the utility function of the secondary user is designed based on the price paid by the secondary user and the quality of communication; and the utility function of the primary user in the game theory model is defined as the reward function of the intelligent agent corresponding to the primary user, and the utility function of the secondary user is defined as the reward function of the intelligent agent corresponding to the secondary user.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the communication resource allocation method of the communication system as described in the above embodiment.

[0019] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the communication resource allocation method of the communication system as described in the above embodiment.

[0020] Therefore, this application has at least the following beneficial effects:

[0021] By inputting the environmental parameters of primary and secondary users from the environmental parameter information of each time slot in the communication system into a pre-trained allocation model, the user's resource allocation strategy is determined. This effectively solves the problem of allocating communication resources in the communication system, enabling each user to formulate a communication resource allocation strategy that maximizes their own benefits, thereby effectively increasing user income and improving communication system performance and communication resource utilization. This solves the problem that the dynamic intelligent allocation of multiple communication resources in related mobile communication systems cannot maximize user benefits and suffers from low communication system performance.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 This is a flowchart of a communication resource allocation method for a communication system provided according to an embodiment of the present application;

[0025] Figure 2 An overall structural diagram of the model relationships and the interactions between various parts of the model provided according to an embodiment of the present application;

[0026] Figure 3This is a block diagram of a communication resource allocation device for a communication system provided according to an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application.

[0028] Description of reference numerals: first acquisition module 100, identification module 200, allocation module 300, memory 401, processor 402, communication interface 403. DETAILED DESCRIPTION

[0029] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes the communication resource allocation method, device, electronic device and storage medium of the communication system of the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a communication resource allocation method for a communication system. In this method, by inputting the environmental parameters of the main user and the environmental parameters of the secondary user in the environmental parameter information of each time slot in the communication system into a pre-trained allocation model, the user's resource allocation strategy is determined, which can effectively solve the problem of allocating communication resources in the communication system, so that each user can formulate a communication resource allocation strategy that maximizes its own benefits, thereby effectively increasing user income and improving the performance of the communication system and the utilization rate of communication resources. As a result, it solves the problems that the dynamic intelligent allocation of multiple communication resources in the mobile communication system of the related art cannot maximize the user's benefits and there is a low performance of the communication system.

[0031] Specifically, Figure 1 A flow chart of a communication resource allocation method for a communication system provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the communication resource allocation method of the communication system includes the following steps:

[0033] In step S101, environmental parameter information of each time slot in the communication system is obtained.

[0034] In this embodiment, the communication system and other users' historical actions can be defined as the environment. First, the environmental parameter information of each time slot in the communication system can be obtained to be applied to subsequent deep reinforcement learning training to complete the decision.

[0035] In step S102 , the environmental parameters of the primary user and the environmental parameters of the secondary users in the environmental parameter information are identified.

[0036] It is understandable that each user is both an intelligent agent and, relative to other users, part of the environment. Embodiments of the present application can identify the environmental parameters of the primary user, train and make decisions using a deep reinforcement learning algorithm, and determine the interference price for each secondary user's access. They can also identify the environmental parameters of the secondary user, train and make decisions using a deep reinforcement learning algorithm, and determine the channel and power for the secondary user's access.

[0037] In step S103, the primary user's environmental parameters and the secondary user's environmental parameters are respectively input into the pre-trained allocation model, and the communication resource allocation strategies for the primary user and the secondary user are output, and the communication resources of the communication system are allocated using the communication resources.

[0038] It can be understood that the embodiments of the present application can input the environmental parameters of the primary user and the environmental parameters of the secondary user into the trained allocation model respectively, make decisions based on the network output results, combine game theory with deep reinforcement learning, and comprehensively utilize their advantages to perform dynamic and intelligent allocation of communication resources.

[0039] Specifically, the primary user and the secondary user input the system state s into the artificial neural network at each moment. The Q value corresponding to all optional actions a of the neural network is Q(s,a). After training, the action strategy of the primary user and the secondary user is:

[0040] a′=argmax a Q(s,a)

[0041] Where a′ is the action selected by the user at that moment. The primary user sets the interference price for each secondary user based on the value of a′. The secondary user then decides whether to access, the channel to access, and the transmission power based on the value of a′.

[0042] Therefore, by selecting appropriate game theory models and reinforcement learning models to model the communication system, rationally designing artificial neural networks and utility functions, and applying deep reinforcement learning technology, the problem of communication resource allocation in the communication system can be effectively solved, enabling each user to formulate a communication resource allocation strategy that maximizes their own benefits, thereby effectively increasing user income, improving communication resource utilization and system performance, and has good application prospects.

[0043] In one embodiment of the present application, the allocation model is trained based on the environmental parameter training data of the primary user and the secondary user, including: constructing an artificial neural network structure according to the system model and optimization objectives of the communication system; using the environmental parameters of the primary user as the input of the artificial neural network corresponding to the primary user, and using the environmental parameters of the secondary user as the input of the artificial neural network corresponding to the secondary user; training the artificial neural network structure through a preset deep reinforcement learning algorithm to obtain the allocation model.

[0044] Among them, the hidden layer of the artificial neural network includes but is not limited to one or more of DNN (Deep Neural Networks), CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), ESN (echo state network), LSTM (Long Short Term Memory Network), and GRU (Gate Recurrent Unit); the activation function selection of the artificial neural network includes but is not limited to any one or more of the following functions: ReLU function and its variants, Sigmoid function, tanh function, softmax function, and step function.

[0045] Specifically, the embodiment of the present application can design an appropriate artificial neural network structure based on the system model and optimization objectives, use the environmental parameters of the primary user as the input of the artificial neural network corresponding to the primary user, and use the environmental parameters of the secondary user as the input of the artificial neural network corresponding to the secondary user. The artificial neural network structure is trained using a preset deep reinforcement learning algorithm. Among them, the selection of the preset deep reinforcement learning algorithm includes but is not limited to any one of the following algorithms: deep Q network or its derivative algorithm, DDPG (deep deterministic policy gradient) algorithm or its derivative algorithm, actor-critic algorithm or its derivative algorithm.

[0046] It should be noted that those skilled in the art can select a suitable artificial neural network model, activation function and training algorithm according to actual conditions, and no specific limitations are imposed here.

[0047] In one embodiment of the present application, a preset deep reinforcement learning algorithm includes a deep Q network and a deep recursive Q network. The artificial neural network structure is trained by the preset deep reinforcement learning algorithm to obtain an allocation model, including: using the deep Q network and the deep recursive Q network to generate a DQN network for each user according to the user's demand strategy; in the process of data generation, for each DQN network, the system state is input into the DQN network in each time slot, and the sequence number of the largest item in the network output is used as the alternative action, and a decision is made according to a preset method, wherein the intelligent agent selects the alternative action with a probability of 1-ε and randomly selects an action with a probability of ε; in the process of updating the network parameters, the data generated by the training process is input into the DQN network, and the new network parameters are calculated according to the network update algorithm until the preset conditions are met to obtain the allocation model.

[0048] The selected network update algorithm may be an optimization-based algorithm. The algorithm includes, but is not limited to, any one or more of the following: gradient descent and its derivatives, Newton-Raphson iterative algorithm, heuristic optimization algorithm, etc. The default method may be the ∈-greedy method. Those skilled in the art may select the algorithm based on actual circumstances, and no specific limitation is imposed here.

[0049] Specifically, the embodiment of the present application can use a deep Q network to train the artificial neural network structure. It is understandable that different users of the same type and different types of users require different strategies, so an artificial neural network is generated for each primary user and secondary user. In the process of data generation, for each DQN network, the system state is input into the network in each time slot to obtain the network output, and a decision is made according to the ∈-greedy method. Specifically, the agent has a probability of 1-ε to select the action with the maximum value output by the network, otherwise it randomly selects one from all optional actions. The value of ε decreases as the number of training steps increases. In the actual execution process, the advantage of the ∈-greedy method is that it enables the agent to explore more outside the known state-action binary at the beginning of training, so as to maximize the long-term benefits of the agent. When updating the parameters in the artificial neural network, the user will extract the generated data and input it into the network, and apply the above-mentioned network update algorithm to update the weight parameters in the network.

[0050] In one embodiment of the present application, a frozen target network technology or an experience replay technology is used for training during the training process. If the DQN algorithm using the frozen target network technology is used, the target network and the prediction network of the DQN network are used to calculate the true value and the estimated value of the Q value respectively, and the training loss is calculated based on the true value and the estimated value. When the training loss is less than the preset loss, the iterative training is stopped. If the DQN algorithm using the experience replay technology is used, the data generated during training is stored in a preset memory. When updating the neural network parameters, the intelligent agent randomly extracts a portion of the data from the preset memory and scrambles it, and applies the network update algorithm for training.

[0051] It should be noted that when the artificial neural network selected in the embodiment of the present application contains RNN, LSTM or GRU, the update of its parameters needs to maintain its original order of iterative calculation. Therefore, the intelligent agent randomly extracts a data sequence of a certain length from the preset memory, maintains its original order, and applies the above-mentioned network update algorithm for training.

[0052] The present embodiment can use frozen target network technology and experience replay technology to improve training results. Specifically, the advantage of using frozen target network in the present embodiment is that it reduces the correlation between the target network and the prediction network, stabilizing the training goal; the advantage of using experience replay is that it reuses the generated data and reduces the correlation in the data sequence.

[0053] In one embodiment of the present application, before model training, it also includes: obtaining the system composition, user information, resource type to be allocated and optimization goal of the communication system; matching the game theory model according to the system composition, user information, resource type and optimization goal, and modeling the communication system resource allocation problem according to the game theory model, and designing the utility function of each user; according to the established game theory model and user utility function, the leader and follower in the game theory model are defined as two intelligent agents, the interference price set by the main user for each secondary user is defined as the action of the main user, and the channel to be accessed by the secondary user and the selected access power are defined as the action of the secondary user; the channel and power accessed by each secondary user in the previous time slot are defined as the environmental parameters of the main user, and the interference price set by each main user for a certain secondary user is defined as the environmental parameters of the secondary user.

[0054] In embodiments of the present application, an appropriate game theory model can be selected based on the composition of the communication system, the user's situation, the type of resources to be allocated, and the system's optimization objectives. Users can be modeled as participants in the game, and each user's utility function can be designed based on the system's optimization objectives and each user's needs. Game theory models can include, but are not limited to, one or more of the Stackelburg model and the Cournot model.

[0055] Specifically, the embodiments of the present application can take the Stackelburg model as an example, defining the primary user in the system as the leader in the Stackelburg model and the secondary users in the system as the followers in the Stackelburg model. The embodiments of the present application can design the utility function of the primary user based on the primary user's income and power consumption, and design the utility function of the secondary user based on the price paid by the secondary user and the quality of communication. For example, the utility function of the primary user PU can be designed as follows:

[0056]

[0057] Among them, λ m is the interference price of the mth secondary user, p m is the transmit power of the mth secondary user, f is a monotonically increasing function, and C is a positive real number.

[0058] In the embodiment of the present application, the utility function of the secondary user SU can be designed as:

[0059] U m =QoS m -f(λ m p m )

[0060] Among them, U m represents the utility function of the mth secondary user, QoS m It represents the communication quality of the mth secondary user, which can be measured by one or more indicators including signal to interference plus noise ratio SINR or other indicators, and f is a monotonically increasing function.

[0061] In some cases, embodiments of the present application may define the utility function of the primary user in the game theory model as the reward function of the agent corresponding to the primary user, and define the utility function of the secondary user as the reward function of the agent corresponding to the secondary user.

[0062] Furthermore, embodiments of the present application can define the agents required for reinforcement learning based on the established game theory model and the designed utility functions for each user, that is, the participants in the game theory model are defined as agents. Embodiments of the present application can define the leader and follower in the game theory model as two types of agents, depending on the type of user. The access conditions set by the primary user for the secondary user are defined as the primary user's actions, and the situation in which the secondary user occupies communication resources is defined as the secondary user's actions. The historical actions of other users (including primary and secondary users) can also be defined as the environment.

[0063] In summary, combined Figure 2The overall system structure diagram shown here illustrates the relationship between the game theory model and the reinforcement learning model, as well as the interactions between the various parts of the model. Users are both participants and agents, perceiving information from the environment and generating actions based on it, impacting the environment. The system provides feedback to the agent in the form of rewards, or utility functions.

[0064] The communication resource allocation method of the communication system in the embodiment of the present application is described in detail below through two specific embodiments.

[0065] 1) Example 1

[0066] Based on the composition of the communication system, the user profile, the type of resources to be allocated, and the system's optimization objectives, the Stackelburg model was selected as the game theory model. The resource allocation problem in the communication system was modeled using the Stackelburg game theory model. The primary user in the system was defined as the leader, and the secondary users in the system were defined as followers. The utility function of each user in the communication system was designed. The primary user's utility function is:

[0067]

[0068] Among them, λ m is the interference price of the mth secondary user, g m is the channel gain from the mth secondary user transmitter to the primary user receiver.

[0069] The secondary user utility function is:

[0070] U m =SINR m -λ m p m g m

[0071] Among them, U m represents the utility function of the mth secondary user, SINR m It represents the signal to interference plus noise ratio when the mth secondary user accesses.

[0072] According to the established game theory model and the designed utility function of each user, the users in the communication system, namely the leaders and followers in the game theory model, are defined as two types of intelligent agents; the interference price λ set by the main user for each secondary user is m Defined as the action of the primary user, whether the secondary user accesses the channel is related to the access power p selected when accessing mThe access power of each secondary user in the previous time slot is defined as the environmental parameter of the primary user, and the interference price set by the primary user for a secondary user is defined as the environmental parameter of the secondary user. The utility function of the primary user in the game theory model is defined as the reward function of the primary user, and the utility function of the secondary user is defined as the reward function of the intelligent agent corresponding to the secondary user.

[0073] According to the system model and optimization objectives, a fully connected artificial neural network structure is selected, and a deep reinforcement learning algorithm based on deep Q network is used to train the primary users and secondary users.

[0074] After training is complete, the environmental parameters are fed into the trained network at each time slot, and decisions are made based on the network's output. The primary user sets the interference price for each secondary user based on the network's output. Secondary users then use this output to decide whether to connect, which channel to connect to, and what transmission power to use.

[0075] The parameter list of the embodiments of the present application is as follows, wherein Table 1 is the parameter list of Embodiment 1.

[0076] Table 1

[0077]

[0078] 2) Example 2

[0079] Based on the composition of the communication system, the user profile, the type of resources to be allocated, and the system's optimization objectives, the Stackelburg model was selected as the game theory model. The resource allocation problem in the communication system was modeled using the Stackelburg game theory model. The primary user in the system was defined as the leader, and the secondary users in the system were defined as followers. The utility function of each user in the communication system was designed. The primary user's utility function is:

[0080]

[0081] Among them, U n represents the utility function of the nth primary user, λ m is the interference price of the mth secondary user, g mn is the channel gain from the mth secondary user transmitter to the nth primary user receiver.

[0082] The secondary user utility function is:

[0083] U m =SINR m -λ m p m g m

[0084] Among them, U m represents the utility function of the mth secondary user, SINR m It represents the signal to interference plus noise ratio when the mth secondary user accesses.

[0085] According to the established game theory model and the designed utility function of each user, the users in the communication system, namely the leaders and followers in the game theory model, are defined as two types of intelligent agents; the interference price λ set by the main user for each secondary user is m Defined as the action of the primary user, the channel to be accessed by the secondary user is connected to the selected access power p m The parameters of the primary user are defined as the actions of the secondary user; the channel and power accessed by each secondary user in the previous time slot are defined as the environmental parameters of the primary user; the interference price set by each primary user for a secondary user is defined as the environmental parameters of the secondary user; the utility function of the primary user in the game theory model is defined as the reward function of the primary user, and the utility function of the secondary user is defined as the reward function of the intelligent agent corresponding to the secondary user.

[0086] According to the system model and optimization objectives, a recurrent artificial neural network structure is selected, and a deep reinforcement learning algorithm based on deep Q network is used to train the primary and secondary users.

[0087] After training is complete, the environmental parameters are fed into the trained network at each time slot, and decisions are made based on the network's output. The primary user sets the interference price for each secondary user based on the network's output. Secondary users then use this output to decide whether to connect, which channel to connect to, and what transmission power to use.

[0088] The parameter list of the embodiments of the present application is as follows, wherein Table 2 is the parameter list of Embodiment 2.

[0089] Table 2

[0090]

[0091] According to the communication resource allocation method for a communication system proposed in an embodiment of the present application, users in the communication system are modeled as participants in game theory and agents in reinforcement learning. By inputting the environmental parameters of the primary user and the secondary user in the environmental parameter information of each time slot in the communication system into a pre-trained allocation model, the user's resource allocation strategy is determined. This can effectively solve the problem of allocating communication resources in the communication system, enabling each user to formulate a communication resource allocation strategy that maximizes their own benefits, thereby effectively increasing user income and improving communication system performance and communication resource utilization. This solves the problem that the dynamic intelligent allocation of multiple communication resources in related mobile communication systems cannot maximize user benefits and suffers from low communication system performance.

[0092] Next, a communication resource allocation device for a communication system proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0093] Figure 3 It is a block diagram of a communication resource allocation device of a communication system according to an embodiment of the present application.

[0094] like Figure 3 As shown, the communication resource allocation device 10 of the communication system includes: a first acquisition module 100 , an identification module 200 and an allocation module 300 .

[0095] Among them, the first acquisition module 100 is used to obtain the environmental parameter information of each time slot in the communication system; the identification module 200 is used to identify the environmental parameters of the primary user and the environmental parameters of the secondary user in the environmental parameter information; the allocation module 300 is used to input the environmental parameters of the primary user and the environmental parameters of the secondary user into the pre-trained allocation model respectively, output the communication resource allocation strategy of the primary user and the secondary user, and use the communication resources to allocate the communication resources of the communication system.

[0096] In one embodiment of the present application, the allocation model is trained based on environmental parameter training data of the primary user and the secondary user.

[0097] In one embodiment of the present application, the device 10 of the embodiment of the present application also includes: a training module, which is used to construct an artificial neural network structure based on the system model and optimization objectives of the communication system; use the environmental parameters of the primary user as the input of the artificial neural network corresponding to the primary user, and use the environmental parameters of the secondary user as the input of the artificial neural network corresponding to the secondary user; train the artificial neural network structure through a preset deep reinforcement learning algorithm to obtain an allocation model.

[0098] In one embodiment of the present application, a preset deep reinforcement learning algorithm includes a deep Q network and a deep recursive Q network, and the training module is further used to use the deep Q network and the deep recursive Q network to generate a DQN network for each user according to the user's demand strategy; in the process of data generation, for each DQN network, the system state is input into the DQN network in each time slot, and the sequence number of the largest item in the network output is used as the alternative action, and a decision is made according to a preset method, wherein the intelligent agent selects the alternative action with a probability of 1-ε and randomly selects an action with a probability of ε; in the process of updating the network parameters, the data generated by the training process is input into the DQN network, and the new network parameters are calculated according to the network update algorithm until the preset conditions are met to obtain the allocation model.

[0099] In one embodiment of the present application, the training module is further used to perform training using a frozen target network technology or an experience replay technology during the training process, wherein, if the DQN algorithm using the frozen target network technology is used, the two neural networks of the target network and the prediction network of the DQN network are used to calculate the true value and the estimated value of the Q value respectively, and the training loss is calculated based on the true value and the estimated value until the training loss is less than the preset loss, and the iterative training is stopped; if the DQN algorithm using the experience replay technology is used, the data generated during training is stored in a preset memory, and when updating the neural network parameters, the intelligent agent randomly extracts a portion of the data from the preset memory and scrambles it, and applies the network update algorithm for training.

[0100] In one embodiment of the present application, the apparatus 10 of the embodiment of the present application further includes: a second acquisition module, a processing module and a definition module.

[0101] Among them, the second acquisition module is used to obtain the system composition, user information, resource type to be allocated and optimization goal of the communication system before model training; the processing module is used to match the game theory model according to the system composition, user information, resource type and optimization goal, and model the communication system resource allocation problem according to the game theory model, and design the utility function of each user; the definition module is used to define the leader and follower in the game theory model as two intelligent agents according to the established game theory model and user utility function, define the interference price set by the main user for each secondary user as the action of the main user, and define the channel to be accessed by the secondary user and the selected access power as the action of the secondary user; define the channel and power accessed by each secondary user in the previous time slot as the environmental parameters of the main user, and define the interference price set by each main user for a secondary user as the environmental parameters of the secondary user.

[0102] In one embodiment of the present application, the utility function of the primary user is designed based on the primary user's income and the power consumed; the utility function of the secondary user is designed based on the price paid by the secondary user and the quality of communication; and the utility function of the primary user in the game theory model is defined as the reward function of the intelligent agent corresponding to the primary user, and the utility function of the secondary user is defined as the reward function of the intelligent agent corresponding to the secondary user.

[0103] It should be noted that the above explanation of the embodiment of the communication resource allocation method of the communication system is also applicable to the communication resource allocation device of the communication system of this embodiment, and will not be repeated here.

[0104] According to the communication resource allocation device of the communication system proposed in the embodiment of the present application, the users in the communication system are modeled as participants in game theory and intelligent agents in reinforcement learning. By inputting the environmental parameters of the primary user and the secondary user in the environmental parameter information of each time slot in the communication system into a pre-trained allocation model, the user's resource allocation strategy is determined. This can effectively solve the problem of allocating communication resources in the communication system, enabling each user to formulate a communication resource allocation strategy that maximizes their own benefits, thereby effectively increasing user income and improving communication system performance and communication resource utilization. This solves the problems of the dynamic intelligent allocation of multiple communication resources in the related mobile communication system that cannot maximize user benefits and has low communication system performance.

[0105] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0106] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0107] When the processor 402 executes the program, the communication resource allocation method of the communication system provided in the above embodiment is implemented.

[0108] Furthermore, the electronic device further includes:

[0109] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0110] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0111] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0112] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0113] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0114] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0115] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the communication resource allocation method of the communication system as described above.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0117] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0118] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0119] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0120] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0121] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A communication resource allocation method for a communication system, characterized in that: The following steps are involved: Obtaining environmental parameter information for each time slot in the communication system; Identifying the environmental parameters of the primary user and the environmental parameters of the secondary user in the environmental parameter information; inputting the environmental parameters of the primary user and the environmental parameters of the secondary user into a pre-trained allocation model respectively, outputting a communication resource allocation strategy for the primary user and the secondary user, and using the communication resources to allocate communication resources of the communication system; The allocation model is trained based on environmental parameter training data of primary users and secondary users, and includes: constructing an artificial neural network structure according to the system model and optimization objectives of the communication system; Using the primary user's environmental parameters as input to the artificial neural network corresponding to the primary user, and using the secondary user's environmental parameters as input to the artificial neural network corresponding to the secondary user; The artificial neural network structure is trained by a preset deep reinforcement learning algorithm to obtain the allocation model; Before model training, it also includes: Obtaining the system composition, user information, resource types to be allocated, and system optimization goals of the communication system; Matching a game theory model according to the system composition, the user information, the resource type, and the optimization goal, modeling the communication system resource allocation problem according to the game theory model, and designing a utility function for each user; According to the established game theory model and the user utility function, the leader and follower in the game theory model are defined as two intelligent entities respectively, the interference price set by the main user for each secondary user is defined as the action of the main user, and the channel to be accessed by the secondary user and the selected access power are defined as the actions of the secondary user; the channel and power accessed by each secondary user in the previous time slot are defined as the environmental parameters of the main user, and the interference price set by each main user for a certain secondary user is defined as the environmental parameters of the secondary user.

2. The method according to claim 1, characterized in that The preset deep reinforcement learning algorithm includes a deep Q network and a deep recursive Q network. The artificial neural network structure is trained by the preset deep reinforcement learning algorithm to obtain the allocation model, including: Use deep Q-network and deep recurrent Q-network to generate a DQN network for each user based on the user's demand strategy; During the data generation process, for each DQN network, the system state is input into the DQN network at each time slot, and the sequence number of the largest item in the network output is used as the alternative action. The decision is made according to the preset method, where the agent selects the alternative action with a probability of 1-ε and randomly selects an action with a probability of ε; During the network parameter update process, the data generated during the training process is input into the DQN network, and new network parameters are calculated according to the network update algorithm until the preset conditions are met to obtain the allocation model.

3. The method according to claim 2, characterized in that During the training process, the target network freezing technology or experience replay technology is used for training, wherein, If the DQN algorithm using the frozen target network technology is used, the target network and prediction network of the DQN network are used to calculate the true value and estimated value of the Q value respectively, and the training loss is calculated based on the true value and the estimated value. Iteration training stops when the training loss is less than the preset loss. If the DQN algorithm using experience replay technology is used, the data generated during training is stored in a preset memory. When updating the neural network parameters, the agent randomly extracts a portion of the data from the preset memory and scrambles it, and applies the network update algorithm for training.

4. The method according to claim 1, wherein The utility function of the primary user is designed based on the primary user's income and power consumption; the utility function of the secondary user is designed based on the price paid by the secondary user and the quality of communication; and the utility function of the primary user in the game theory model is defined as the reward function of the intelligent agent corresponding to the primary user, and the utility function of the secondary user is defined as the reward function of the intelligent agent corresponding to the secondary user.

5. A communication resource allocation device for a communication system, characterized in that: The apparatus is used to implement the communication resource allocation method of the communication system according to any one of claims 1 to 4, comprising: A first acquisition module is used to obtain environmental parameter information of each time slot in the communication system; an identification module, configured to identify the environmental parameters of the primary user and the environmental parameters of the secondary user in the environmental parameter information; The allocation module is used to input the environmental parameters of the primary user and the environmental parameters of the secondary user into the pre-trained allocation model respectively, output the communication resource allocation strategy of the primary user and the secondary user, and use the communication resources to allocate the communication resources of the communication system.

6. The device according to claim 5, characterized in that Also includes: A training module is configured to obtain the allocation model by training environmental parameter training data of the primary user and the secondary user, wherein an artificial neural network structure is constructed according to the system model and optimization objectives of the communication system; the environmental parameters of the primary user are used as inputs of the artificial neural network corresponding to the primary user, and the environmental parameters of the secondary user are used as inputs of the artificial neural network corresponding to the secondary user; the artificial neural network structure is trained using a preset deep reinforcement learning algorithm to obtain the allocation model, the preset deep reinforcement learning algorithm including a deep Q network and a deep recursive Q network, and the artificial neural network structure is trained using the preset deep reinforcement learning algorithm. Obtaining the allocation model includes: using a deep Q network and a deep recursive Q network to generate a DQN network for each user according to the user's demand strategy; in a data generation process, for each DQN network, inputting the system state into the DQN network at each time slot, using the sequence number of the largest item in the network output as an alternative action, and making a decision according to a preset method, wherein the intelligent agent selects the alternative action with a probability of 1-ε and randomly selects an action with a probability of ε; in a network parameter update process, inputting data generated in a training process into the DQN network, and calculating new network parameters according to a network update algorithm until preset conditions are met, thereby obtaining the allocation model.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the communication resource allocation method of the communication system according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the communication resource allocation method of the communication system according to any one of claims 1 to 4.

Images