Spectrum access method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a spectrum access method and device, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a maximum return value corresponding to a current user to be accessed; on the basis of the maximum return value and / or the total number of the users currently accessing the multiple spectrums, determining whether the spectrums are allowed to be allocated to the current to-be-accessed users or not; and in response to determining that the spectrum is allowed to be allocated to the current to-be-accessed user, allocating the spectrum corresponding to the maximum return value in the plurality of spectrums to the current to-be-accessed user as a target spectrum. Thus, the return value corresponding to each frequency spectrum in the plurality of frequency spectrums is the return value obtained by assuming that the frequency spectrum is allocated to the current user to be accessed, that is, in the disclosure, each frequency spectrum can be dynamically tried by acquiring the return value corresponding to each frequency spectrum. Therefore, the optimal spectrum capable of balancing the performance in all aspects can be screened through continuous attempts and feedback, so that the efficiency and the fairness of dynamic spectrum management can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a spectrum access method, apparatus, electronic device, storage medium, and computer program product. Background Art

[0002] With the rapid expansion of IoT applications, data transmission is becoming more and more intensive and data computing is becoming more and more demanding, which poses a higher challenge to the allocation of wireless spectrum resources. The traditional fixed spectrum allocation method can no longer meet these growing demands, resulting in a serious waste of spectrum resources in time and space.

[0003] In the related art, a dynamic spectrum access technology is proposed, which allows a device to access the idle spectrum immediately when it detects it, so that unoccupied spectrum resources can be effectively utilized. However, the dynamic spectrum access method in the related art still has many deficiencies in terms of efficiency and fairness of dynamic spectrum management. Summary of the invention

[0004] The present disclosure provides a spectrum access method, apparatus, electronic device, storage medium and computer program product to at least solve the problem that the dynamic spectrum access method in the above-mentioned related technologies still has many shortcomings in terms of efficiency and fairness of dynamic spectrum management.

[0005] According to a first aspect of an embodiment of the present disclosure, a spectrum access method is provided, including: obtaining a maximum reward value corresponding to a current user to be accessed, wherein the maximum reward value is the maximum value of multiple reward values ​​corresponding to multiple spectrums one by one, and the reward value corresponding to each spectrum in the multiple spectrums is a reward value obtained by assuming that the spectrum is allocated to the current user to be accessed; based on the maximum reward value and / or the total number of users who have currently accessed the multiple spectrums, determining whether to allow the allocation of spectrum to the current user to be accessed; in response to determining that the allocation of spectrum to the current user to be accessed is allowed, allocating the spectrum corresponding to the maximum reward value in the multiple spectrums as the target spectrum to the current user to be accessed.

[0006] Optionally, the determining whether to allow spectrum allocation for the current user to access includes: calculating the difference between the maximum reward value corresponding to the current user to access and the maximum reward value corresponding to the previous user to access; in response to the difference being greater than or equal to a preset threshold, determining that spectrum allocation for the current user to access is allowed.

[0007] Optionally, the determination of whether to allow spectrum allocation for the current user to access includes: estimating the estimated total number of users who have currently accessed the multiple spectra when spectrum is allocated to the current user to access; comparing the estimated total number with a preset number threshold; in response to the estimated total number being less than or equal to the preset number threshold, determining that spectrum allocation for the current user to access is allowed.

[0008] Optionally, the spectrum access method further includes: in response to allocating the target spectrum to the current user to be accessed, adding one to the number of times the target spectrum is allocated, wherein the number of times the target spectrum is allocated is used to calculate a reward value corresponding to the target spectrum.

[0009] Optionally, the spectrum access method further includes: in response to a user currently occupying the target spectrum releasing occupation of the target spectrum, reducing the number of times the target spectrum is allocated by one.

[0010] According to a second aspect of an embodiment of the present disclosure, a spectrum access device is provided, including: a reward value acquisition module, configured to acquire a maximum reward value corresponding to a current user to be accessed, wherein the maximum reward value is the maximum value of a plurality of reward values ​​corresponding to a plurality of spectrums one by one, and the reward value corresponding to each spectrum in the plurality of spectrums is a reward value obtained by assuming that the spectrum is allocated to the current user to be accessed; a spectrum determination module, configured to determine whether to allow the allocation of spectrum to the current user to be accessed based on the maximum reward value and / or the total number of users who have currently accessed the plurality of spectrums; an allocation module, configured to allocate the spectrum corresponding to the maximum reward value in the plurality of spectrums as the target spectrum to the current user to be accessed in response to determining that the allocation of spectrum to the current user to be accessed is allowed.

[0011] Optionally, the spectrum determination module is configured to: calculate the difference between the maximum reward value corresponding to the current user to be connected and the maximum reward value corresponding to the previous user to be connected; in response to the difference being greater than or equal to a preset threshold, determine to allow spectrum to be allocated to the current user to be connected.

[0012] Optionally, the spectrum determination module is configured to: estimate the estimated total number of users who have currently accessed the multiple spectra when allocating spectrum to the current user to be accessed; compare the estimated total number with a preset number threshold; in response to the estimated total number being less than or equal to the preset number threshold, determine to allow allocation of spectrum to the current user to be accessed.

[0013] Optionally, the spectrum access device also includes: an allocation times increasing module, configured to increase the number of allocations of the target spectrum by one in response to allocating the target spectrum to the current user to be accessed, wherein the number of allocations is used to calculate a reward value corresponding to the target spectrum.

[0014] Optionally, the spectrum access device further includes: an allocation times reduction module configured to reduce the number of allocation times of the target spectrum by one in response to a user currently occupying the target spectrum releasing occupancy of the target spectrum.

[0015] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the spectrum access method according to the present disclosure.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the spectrum access method according to the present disclosure.

[0017] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the spectrum access method according to the present disclosure when executed by a processor.

[0018] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: In the present disclosure, the reward value corresponding to each spectrum in the multiple spectrums is the reward value obtained by assuming that the spectrum is allocated to the current waiting user, that is, in the present disclosure, each spectrum can be dynamically tried by obtaining the reward value corresponding to each spectrum. In this way, through continuous attempts and feedback, the best spectrum that can balance various performances can be screened, thereby improving the efficiency and fairness of dynamic spectrum management.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 Detailed Description of the Invention FIG. 1 is a schematic diagram illustrating a configuration and mutual relationship of a wireless communication system according to an exemplary embodiment of the present disclosure.

[0022] Figure 2is a flowchart illustrating a spectrum access method according to an exemplary embodiment of the present disclosure.

[0023] Figure 3 is a schematic diagram illustrating a “user-spectrum matrix” according to an exemplary embodiment of the present disclosure.

[0024] Figure 4 is a flowchart showing a specific implementation of a spectrum access method according to an exemplary embodiment of the present disclosure.

[0025] Figure 5 is a block diagram illustrating a spectrum access apparatus according to an exemplary embodiment of the present disclosure.

[0026] Figure 6 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0029] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0030] The Internet of Things has excellent connectivity and data processing capabilities. However, with the popularity of its applications, wireless spectrum resources have become increasingly scarce. For example, the spectrum access process is prone to large access delays, high energy consumption, and even unavailable access. In addition, with the rapid expansion of IoT applications (for example, with the development of augmented / virtual reality network games, the development of autonomous driving and intelligent everything, and the rapid growth of mobile devices and wireless communication applications), there is an urgent need for intensive data transmission, high computing requirements, and low latency, which poses a higher challenge to the reasonable allocation of wireless spectrum resources. The traditional fixed spectrum allocation method has been unable to meet these growing demands, resulting in a serious waste of spectrum resources in time and space.

[0031] At present, dynamic spectrum access technology has become the key to improving communication quality and network efficiency, especially in the field of Internet of Things. This technology allows devices to access the idle spectrum immediately when it detects it, thereby effectively utilizing unoccupied spectrum resources. However, the dynamic spectrum access method in related technologies still has shortcomings in terms of convergence, learning efficiency and fairness.

[0032] In order to solve the above problems existing in the related art, the spectrum access method, device, electronic device, storage medium and computer program product provided by the present disclosure, the reward value corresponding to each spectrum in the multiple spectrums is the reward value obtained by assuming that the spectrum is allocated to the current user to be accessed, that is, in the present disclosure, each spectrum can be dynamically tried by obtaining the reward value corresponding to each spectrum. In this way, through continuous attempts and feedback, the best spectrum that can balance various performances can be screened, thereby improving the efficiency and fairness of dynamic spectrum management.

[0033] It should be noted that, as an important branch of the field of reinforcement learning, reinforcement learning technology is a key method to solve the problem of wireless spectrum access and channel selection. This technology can optimize the decision-making process through continuous attempts and feedback. In this regard, the present disclosure provides an IoT communication system model covering M users and K spectrums, where the number of users M is greater than or equal to the number of spectrums K.

[0034] Figure 1 Detailed Description of the Invention FIG. 1 is a schematic diagram illustrating a configuration and mutual relationship of a wireless communication system according to an exemplary embodiment of the present disclosure.

[0035] Reference Figure 1The IoT communication system model takes into account the dynamics of the spectrum, the user's access requirements, and channel interference and fading. In the IoT environment, the behavior of users generating service requirements is random, and a part of them can be randomly selected from all users. These randomly selected users follow a uniform distribution. The design of the IoT communication system model can fairly select different users to truly simulate the demand generation scenario of the IoT. Whenever a spectrum access demand is generated, the relevant user requesting access to the spectrum will be regarded as an active user, and the IoT communication system model needs to select a suitable access spectrum for the active user. Specifically, time management can be further subdivided. For example, T time slots can be considered, and there may be multiple users in each time slot who need to access the spectrum. In addition, each time slot can be divided into t time frames, and each time frame can allow one active user to access the spectrum. The number of time frames depends on the number of access users.

[0036] It should be noted that if multiple active users access the same spectrum in the same time slot, they will have mutual influence on each other, namely spectrum interference, which will have a negative impact on the data transmission rate. In order to optimize this process, the present disclosure uses classic spectrum interference elimination technology to ensure smooth data transmission. In addition, based on Figure 1 The wireless communication system model shown in the figure establishes a spectrum access mathematical model, which can maximize the data transmission rate of the network by optimizing the spectrum access variables of the users.

[0037] When dynamic spectrum access is performed in the Internet of Things, each decision time slot may involve the situation where multiple users need to access the appropriate spectrum at the same time for data transmission, that is, each decision time slot may have to handle multiple spectrum selections at the same time. In this regard, the present disclosure introduces reinforcement learning, and the agent can select a set of spectrums in each time frame. At this time, the dynamic spectrum access problem is modeled as a Markov decision problem for a single agent, where each spectrum can be modeled as a discrete-time, irreducible and non-periodic Markov chain with a finite state space. Moreover, in each decision time slot, the agent can observe the states of all actions and can select appropriate actions for multiple users to access. In addition, each choice involves an unknown probability distribution, and the agent needs to learn these distributions through repeated selection experiments in order to optimize its benefits.

[0038] However, reinforcement learning to solve a set of spectrum access problems faces huge challenges: 1) Large-scale action space: The agent can select multiple spectrums at the same time. When the number of spectrums increases, the action space will rapidly expand. 2) Complex reward function: Since the selected actions are dependent on and influence each other, the reward function is usually not a linear sum of the rewards for each action, but may be a more complex function. 3) Constraints of wireless communication resources: The decision of the agent will be affected by the spectrum occupancy, that is, it is necessary to make the optimal spectrum selection under the condition of satisfying resource constraints.

[0039] In response to the above challenges, this disclosure proposes a nested reinforcement learning method, which can be mainly divided into two parts: "inner loop" and "outer loop". Figure 1 In the "outer loop", the agent observes the external environment and obtains a set of activated users. In each "inner loop" decision, the confidence upper bound algorithm is extended by introducing a submodular function to assist activated users in accessing suitable spectrum. By utilizing the submodular function based on the data transmission rate, the agent selects suitable access spectrum for activated users in turn. In addition, the number of activated users to whom spectrum is allocated is not yet known. Some activated users may not be allocated suitable spectrum. In this case, these activated users can remain activated in the next round of "inner loop" to wait for spectrum allocation.

[0040] It should be noted that the conditions of each spectrum are time-varying (channel fading phenomenon), so the spectrum access problem is an unstable reinforcement learning problem. Each spectrum can be modeled as an arbitrary finite state discrete time Markov chain. In addition, Markov decision usually includes "state", "action", "reward", "state transition probability", "regret value" and so on. Specifically, the Markov decision model can be as follows: "State": For the dynamic spectrum access problem, the state information of each action may include: spectrum occupancy, users to be accessed, user data transmission requirements, etc. In this disclosure, it is assumed that these actions (or Markov chains) are independent of each other and unstable, that is, their states will change with channel fading.

[0041] “Action”: Select the appropriate spectrum for activating the user.

[0042] "Reward": In traditional reinforcement learning problems, each "action" corresponds to a potential reward, and the distribution of these rewards is fixed throughout the process. However, in unstable reinforcement learning problems, the reward distribution changes over time or changes with channel fading, which greatly increases the complexity of the problem. The state of each Markov chain can evolve independently of the user's actions, and the reward for an activated user is a function of the spectrum state or the Markov chain state, which can reflect the user's data transmission rate. In addition, due to the influence of interference between users within the system, the reward function for a group of activated users is not the linear sum of all user data transmission rates, but the total data transmission rate of the activated users after the activated users simultaneously access the spectrum in the current time slot.

[0043] "State transition probability": activated users access the spectrum in sequence, and the decisions of the users will affect each other. In this regard, the present disclosure divides a decision time slot into t time frames and introduces the state transition probability, which can be close to the Markov decision process. The state transition probability defines the probability of going from the current state to the new state, which depends on the action taken.

[0044] "Regret value": For spectrum access strategies, the regret can be defined as the difference between the expected total reward that can be obtained through the action with the highest average reward in a time slot and the expected total reward obtained by using spectrum access. And the action with the highest average reward can be called the "optimal action"; correspondingly, other actions can be called "suboptimal actions".

[0045] Next, we will combine Figure 2~Figure 6 To specifically illustrate the specific implementation process of the spectrum access method provided by the present disclosure.

[0046] Figure 2 is a flowchart illustrating a spectrum access method according to an exemplary embodiment of the present disclosure.

[0047] Reference Figure 2 In step 201, the maximum reward value corresponding to the current user to be connected can be obtained, wherein the maximum reward value can be the maximum value of multiple reward values ​​corresponding to multiple spectrums one by one, and the reward value corresponding to each spectrum in the multiple spectrums can be the reward value obtained by assuming that the spectrum is allocated to the current user to be connected.

[0048] For example, the reward value corresponding to the spectrum can be calculated by the following formula: Formula (1) Formula (2) in, For each spectrum corresponding to the reward value, , that is, f(X) is the submodular function corresponding to the spectrum, X is the "user-spectrum matrix" obtained by assuming that the spectrum is allocated to the current user to be connected, is the number of times the spectrum has been selected, that is, is the number of times the spectrum has been currently allocated, t is the access order of the current user to be accessed, that is, t is the current time frame, B is the channel bandwidth (in Hertz Hz), S is the signal power (in Watt W), N is the noise power (in Watt W), It is the signal-to-noise ratio (SNR), which is usually used to evaluate the quality of the signal.

[0049] In addition, the “user-spectrum matrix” X can be a two-dimensional matrix composed of activated users and spectra. For example, assuming that the number of activated users is M and the total number of spectra is K, the “user-spectrum matrix” X can be a (M ) size. Moreover, when a certain active user is allocated a certain spectrum, the value of the element at the corresponding position in the matrix can be 1; otherwise, the value of the element at the corresponding position can be 0.

[0050] Figure 3 is a schematic diagram showing a “user-spectrum matrix” according to an exemplary embodiment of the present disclosure. Figure 3 , assuming that the number of activated users M=8, namely: activated user A, activated user B, activated user C, activated user D, activated user E, activated user F, activated user G, activated user S; assuming that the total number of spectrum K=6, namely: spectrum a, spectrum b, spectrum c, spectrum d, spectrum e, spectrum g. Then these activated users and spectrums can form an M =8 A two-dimensional matrix.

[0051] Reference Figure 3 , assuming that the current user to be connected is activated user C, and assuming that spectrum b was previously allocated to activated user A and spectrum c was allocated to activated user B. Then the value of the element X(1,2) at the first row and second column in the "user-spectrum matrix" X can be 1, representing "spectrum b is allocated to activated user A"; the value of the element X(2,3) at the second row and third column in the "user-spectrum matrix" X can be 1, representing "spectrum c is allocated to activated user B".

[0052] Moreover, the values ​​of the other elements except the element X(1, 2) in the first row of the "user-spectrum matrix" X can all be 0, indicating that no corresponding spectrum is allocated to activated user A; similarly, the values ​​of the other elements except the element X(2, 3) in the second row of the "user-spectrum matrix" X can all be 0, indicating that no corresponding spectrum is allocated to activated user B. In addition, the values ​​of the 6 elements in each row from the third row to the eighth row of the "user-spectrum matrix" X can all be 0, indicating that currently no spectrum has been allocated to activated user C, activated user D, activated user E, activated user F, activated user G, and activated user S.

[0053] Thus, in step 201, illustratively, the maximum reward value corresponding to the current user to be accessed: C can be obtained, and the maximum reward value can be the maximum value of the six reward values ​​corresponding to the six spectrums: spectrum a, spectrum b, spectrum c, spectrum d, spectrum e, and spectrum g. In addition, the reward value corresponding to each of the six spectrums can be the reward value obtained by assuming that the spectrum is assigned to the current user to be accessed: C. It should be noted that in the present disclosure, "the reward value corresponding to each spectrum" can specifically refer to the reward value obtained by the intelligent agent after assuming that the spectrum is assigned to the current user to be accessed.

[0054] It should be noted that, assuming that a certain spectrum is allocated to the current user to be connected, the value of the element at the corresponding position in the "user-spectrum matrix" X can be 1. For example, assuming that the spectrum a is allocated to the current user to be connected: C, the value of the element at the position X(3,1) in the 3rd row and 1st column in the "user-spectrum matrix" X can be changed from 0 to 1. At this time, it can be calculated based on formula (2): ; In addition, it is also necessary to determine the number of times spectrum a has been currently allocated In addition, since the access order of the current user to be accessed: C is 3, the value of t is 3 at this time. In this way, the return value corresponding to spectrum a can be calculated using formula (1): .

[0055] Similarly, assuming that the spectrum b is allocated to the current user to be connected: C, the value of the element at the 3rd row and 2nd column in the "user-spectrum matrix" X: X(3, 2) can change from 0 to 1. At this time, it can be calculated based on formula (2): ; In addition, it is also necessary to determine the number of times spectrum b has been currently allocated In addition, since the access order of the current user to be accessed: C is 3, the value of t is 3 at this time. In this way, the return value corresponding to spectrum b can be calculated using formula (1): .

[0056] By analogy, the reward value corresponding to each spectrum when each of the six spectrums is allocated to the current user to be connected: C can be calculated. Then, the maximum reward value among the six reward values ​​obtained when each of the six spectrums is allocated to the current user to be connected: C can be determined.

[0057] In step 202, it may be determined whether to allow the spectrum to be allocated to the current user to be accessed based on the maximum reward value and / or the total number of users who have currently accessed multiple spectrums.

[0058] According to an exemplary embodiment of the present disclosure, the difference between the maximum reward value corresponding to the current user to be connected and the maximum reward value corresponding to the previous user to be connected can be calculated. Then, in response to the difference being greater than or equal to a preset threshold, it can be determined that spectrum allocation for the current user to be connected is allowed; otherwise, in response to the difference being less than the preset threshold, it can be determined that spectrum allocation for the current user to be connected is prohibited.

[0059] In this way, when the difference between the maximum reward value corresponding to the current user to be accessed and the maximum reward value corresponding to the previous user to be accessed is greater than or equal to the preset threshold, it means that the reward obtained by accessing the spectrum for the current user to be accessed is relatively large. Therefore, at this time, it can be allowed to allocate spectrum to the current user to be accessed, thereby ensuring the rationality and fairness of dynamic spectrum management.

[0060] According to an exemplary embodiment of the present disclosure, the estimated total number of users who have currently accessed multiple spectrums when allocating spectrum to the current user to be accessed can be estimated. Then, the estimated total number can be compared with a preset number threshold. Next, in response to the estimated total number being less than or equal to the preset number threshold, it can be determined that the spectrum is allowed to be allocated to the current user to be accessed; otherwise, in response to the estimated total number being greater than the preset number threshold, it can be determined that the spectrum is prohibited to be allocated to the current user to be accessed.

[0061] In this way, when the estimated total number of users who have currently accessed multiple spectrums is less than or equal to the preset number threshold, it means that the number of users who have currently occupied the spectrum is still relatively small, that is, it means that there is still room for allocating spectrum to new users. Therefore, at this time, spectrum can be allocated to the current users to be accessed, thereby ensuring the rationality and fairness of dynamic spectrum management.

[0062] In step 203, in response to determining that it is allowed to allocate spectrum to the current user to be connected, a spectrum corresponding to the maximum reward value among the multiple spectrums may be allocated to the current user to be connected as a target spectrum.

[0063] According to an exemplary embodiment of the present disclosure, in response to allocating the target spectrum to the current waiting user, the number of times the target spectrum is allocated may be increased by one, wherein the number of times the target spectrum is allocated is the number mentioned above. , which can be used to calculate the return value corresponding to the target spectrum. In this way, after each spectrum is allocated, the number of times the spectrum is allocated can be accumulated and added by one, that is, the spectrum occupancy status can be updated in time based on the spectrum occupancy, which is convenient for providing reference when allocating spectrum to new users in the future, thereby ensuring the rationality of subsequent spectrum allocation.

[0064] According to an exemplary embodiment of the present disclosure, in response to a user currently occupying the target spectrum releasing occupation of the target spectrum, the number of times the target spectrum has been allocated can be reduced by one. In this way, when each occupied spectrum is released, the number of times the spectrum has been allocated can be reduced by one, that is, the occupancy status of the spectrum can be updated in a timely manner based on the occupancy of the spectrum, so as to provide a reference for subsequent allocation of spectrum to new users, thereby ensuring the rationality of subsequent spectrum allocation.

[0065] In the present disclosure, the access problem of dynamic spectrum in the Internet of Things is modeled as a reinforcement learning problem, that is, the present disclosure proposes a spectrum access method of nested reinforcement learning, which can mainly include two parts: an inner loop and an outer loop. Specifically, activated users to be accessed are generated in the outer loop; and the specific access spectrum of the activated users is given in the inner loop. Moreover, in the inner loop, an upper confidence bound expansion method based on the marginal diminishing effect is adopted to balance exploration and utilization, and a multi-step decision-making process is used to adapt to the dynamic changes of the spectrum, which can improve the spectrum utilization rate and optimize the data transmission performance, thereby improving the efficiency and fairness of dynamic spectrum management.

[0066] Figure 4 is a flowchart showing a specific implementation of a spectrum access method according to an exemplary embodiment of the present disclosure.

[0067] Reference Figure 4 In step 401, initialization is first performed.

[0068] Specifically, within a time slot, the agent can select a combined action (that is, select a set of appropriate access spectrum for a set of activated users). First, the time frame can be initialized to 0, the set of activated users that have been accessed can be initialized to be empty, the occupied spectrum combination can be initialized to be empty, and the average return of the occupied spectrum combination can be initialized to 0. In addition, the following steps can be cyclically performed within each time frame: an activated user is randomly selected from all activated users, and this activated user can be added to the "accessed activated user set".

[0069] In step 402, the acquisition function collects.

[0070] Specifically, submodular functions can be introduced as acquisition functions in reinforcement learning to handle dynamic spectrum access problems with combined action spaces and complex dependencies. Submodular functions are suitable for combined action selection in reinforcement learning due to their own diminishing marginal returns, so that they can effectively balance the exploration and utilization of the model. The data transmission rate of the accessed user can be defined as a submodular function f, and any set of occupied spectrum can be mapped to the data transmission rate of an accessed user. For any two subsets A and B in the occupied spectrum set S, A is a subset of B, when any spectrum x in S is not in B, the function f satisfies: the data transmission rate brought by the spectrum x added to the set A is greater than or equal to the data transmission rate brought by the spectrum x added to the set B, then this function has submodular characteristics. When the marginal gain threshold or the maximum number of iterations is reached, new users can be stopped from accessing in this time slot.

[0071] In the present disclosure, the classic upper confidence bound acquisition function can be extended, and then an acquisition function based on the marginal diminishing effect can be proposed, which can mainly include two stages: "utilization" and "exploration". Specifically, the calculation of the upper confidence bound can be expressed as:

[0072] in, For each spectrum corresponding to the reward value, , that is, f(X) is the submodular function corresponding to the spectrum, X is the "user-spectrum matrix" obtained by assuming that the spectrum is allocated to the current user to be connected, is the number of times the spectrum has been selected, that is, is the number of times the spectrum has been currently allocated, t is the access order of the current users to be accessed, that is, t is the current time frame. It is an exploration term, which is an uncertainty measure that encourages the system to choose actions that have not been fully explored. At the same time, the upper confidence bound calculation can also include dynamic adjustment parameters, which can dynamically adjust the balance between "exploration" and "exploitation" according to the fluctuation of the model.

[0073] It should be noted that for each combined action (i.e., the selected spectrum combination), the upper confidence bound value can be calculated using the acquisition function as the confidence upper bound of the action, rather than calculating the acquisition function for a single action. The activated user can choose the upper confidence bound value of this spectrum combination to be the current estimated reward (utilization) plus an additional "confidence range" (exploration). The marginal diminishing function is incorporated into the reward value obtained by the currently selected spectrum; the "confidence range" is an increasing value that reflects the compensation for the uncertainty of the reward. Its role is to increase the exploration ability of the model, so that the spectrum that has not been fully explored also has the opportunity to be accessed by the activated user. In addition, in the stage with larger rewards, it can be inclined to "explore", and in the stage with smaller fluctuations, it can be inclined to "utilize", so as to improve the overall performance.

[0074] In step 403, a spectrum is selected. Specifically, the agent may select a spectrum with the highest upper confidence limit value for a given user to access, and may add the allocated spectrum to the "occupied spectrum combination".

[0075] In step 404, the statistics are updated. Specifically, the average reward estimate and the number of selections of each selected action may be updated, that is, the average reward estimate and the number of selections of each selected spectrum may be updated.

[0076] In step 405, it is determined whether to terminate the decision. Specifically, as described above, when the marginal gain threshold or the maximum number of iterations is reached, the decision can be determined to be terminated, that is, it can be determined that new users need to be stopped from being admitted in the current time slot; otherwise, the process can return to step 402. The iteration loop is performed in this way until it is determined that the decision can be terminated.

[0077] It should be noted that due to the dynamic uncertainty of the wireless communication environment of the Internet of Things, the spectrum occupancy and the channel conditions of data transmission will change dynamically over time. Therefore, the nested reinforcement learning algorithm needs to continuously learn and adapt to the changing environment to update the observed spectrum occupancy. Specifically, in each decision time slot, the appropriate spectrum can be selected for multiple activated users to communicate based on the current learning and observation. In addition, the spectrum occupancy and rewards can also be recorded for future decision-making. In addition, the performance indicators of the spectrum access strategy can be regularly evaluated and adjusted to improve spectrum utilization efficiency and network performance.

[0078] Figure 5 is a block diagram illustrating a spectrum access apparatus 500 according to an exemplary embodiment of the present disclosure.

[0079] Reference Figure 5 The spectrum access device 500 may include a reward value acquisition module 501, a spectrum determination module 502 and an allocation module 503.

[0080] The reward value acquisition module 501 can acquire the maximum reward value corresponding to the current user to be connected, wherein the maximum reward value can be the maximum value of multiple reward values ​​corresponding to multiple spectrums one by one, and the reward value corresponding to each spectrum in the multiple spectrums can be the reward value obtained by assuming that the spectrum is allocated to the current user to be connected. It should be noted that in the present disclosure, "the reward value corresponding to each spectrum" can specifically refer to the reward value obtained by the agent after assuming that the spectrum is allocated to the current user to be connected.

[0081] The spectrum determination module 502 may determine whether to allow the spectrum to be allocated to the current user to be accessed based on the maximum reward value and / or the total number of users who have currently accessed multiple spectrums.

[0082] According to an exemplary embodiment of the present disclosure, the spectrum determination module 502 may calculate the difference between the maximum reward value corresponding to the current user to be connected and the maximum reward value corresponding to the previous user to be connected. Then, in response to the difference being greater than or equal to a preset threshold, the spectrum determination module 502 may determine that the spectrum is allowed to be allocated to the current user to be connected; otherwise, that is, in response to the difference being less than the preset threshold, the spectrum determination module 502 may determine that the spectrum is prohibited to be allocated to the current user to be connected.

[0083] In this way, when the difference between the maximum reward value corresponding to the current user to be accessed and the maximum reward value corresponding to the previous user to be accessed is greater than or equal to the preset threshold, it means that the reward obtained by accessing the spectrum for the current user to be accessed is relatively large. Therefore, at this time, it can be allowed to allocate spectrum to the current user to be accessed, thereby ensuring the rationality and fairness of dynamic spectrum management.

[0084] According to an exemplary embodiment of the present disclosure, the spectrum determination module 502 may estimate the estimated total number of users who have currently accessed multiple spectrums when allocating spectrum to the current user to be accessed. Then, the spectrum determination module 502 may compare the estimated total number with a preset number threshold. Next, in response to the estimated total number being less than or equal to the preset number threshold, the spectrum determination module 502 may determine that the spectrum is allowed to be allocated to the current user to be accessed; otherwise, that is, in response to the estimated total number being greater than the preset number threshold, the spectrum determination module 502 may determine that the spectrum is prohibited to be allocated to the current user to be accessed.

[0085] In this way, when the estimated total number of users who have currently accessed multiple spectrums is less than or equal to the preset number threshold, it means that the number of users who have currently occupied the spectrum is still relatively small, that is, it means that there is still room for allocating spectrum to new users. Therefore, at this time, spectrum can be allocated to the current users to be accessed, thereby ensuring the rationality and fairness of dynamic spectrum management.

[0086] In response to determining that it is allowed to allocate spectrum to the current user to be connected, the allocation module 503 may allocate the spectrum corresponding to the maximum reward value among the multiple spectrums as the target spectrum to the current user to be connected.

[0087] According to an exemplary embodiment of the present disclosure, the spectrum access device 500 may further include an allocation times increasing module. In response to allocating the target spectrum to the current access user, the allocation times increasing module may increase the allocation times of the target spectrum by one, wherein the allocation times is the aforementioned , which can be used to calculate the return value corresponding to the target spectrum. In this way, after each spectrum is allocated, the number of times the spectrum is allocated can be accumulated and added by one, that is, the spectrum occupancy status can be updated in time based on the spectrum occupancy, which is convenient for providing reference when allocating spectrum to new users in the future, thereby ensuring the rationality of subsequent spectrum allocation.

[0088] According to an exemplary embodiment of the present disclosure, the spectrum access device 500 may further include an allocation times reduction module. The allocation times reduction module may reduce the number of times the target spectrum is allocated by one in response to the user currently occupying the target spectrum releasing the occupation of the target spectrum. In this way, when each occupied spectrum is released, the number of times the spectrum is allocated may be reduced by one, that is, the spectrum occupation status may be updated in time based on the spectrum occupation status, so as to provide a reference for the subsequent allocation of spectrum to new users, thereby ensuring the rationality of subsequent spectrum allocation.

[0089] Figure 6 is a block diagram illustrating an electronic device 600 according to an exemplary embodiment of the present disclosure.

[0090] Reference Figure 6 The electronic device 600 includes at least one memory 601 and at least one processor 602, wherein the at least one memory 601 stores instructions, and when the instructions are executed by the at least one processor 602, the spectrum access method according to the exemplary embodiment of the present disclosure is executed.

[0091] As an example, the electronic device 600 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 600 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 600 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0092] In the electronic device 600, the processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0093] The processor 602 may execute instructions or codes stored in the memory 601, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0094] The memory 601 may be integrated with the processor 602, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 601 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 601 and the processor 602 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 602 can read files stored in the memory.

[0095] In addition, the electronic device 600 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 600 may be connected to each other via a bus and / or a network.

[0096] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the above-mentioned spectrum access method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0097] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the spectrum access method according to the present disclosure is implemented.

[0098] According to the spectrum access method, device, electronic device, storage medium and computer program product disclosed in the present invention, the reward value corresponding to each spectrum in the multiple spectrums is the reward value obtained by assuming that the spectrum is allocated to the current user to be accessed, that is, in the present invention, each spectrum can be dynamically tried by obtaining the reward value corresponding to each spectrum. In this way, through continuous attempts and feedback, the best spectrum that can balance various performance aspects can be screened, thereby improving the efficiency and fairness of dynamic spectrum management.

[0099] According to an exemplary embodiment of the present disclosure, when the difference between the maximum reward value corresponding to the current user to be accessed and the maximum reward value corresponding to the previous user to be accessed is greater than or equal to a preset threshold, it means that the reward obtained by accessing the spectrum for the current user to be accessed is relatively large. Therefore, at this time, it can be allowed to allocate spectrum to the current user to be accessed, thereby ensuring the rationality and fairness of dynamic spectrum management.

[0100] According to an exemplary embodiment of the present disclosure, when the estimated total number of users who have currently accessed multiple spectrums is less than or equal to a preset number threshold, it means that the number of users who have currently occupied the spectrum is still relatively small, that is, it means that there is still room for allocating spectrum to new users. Therefore, at this time, it is allowed to allocate spectrum to the current user to be accessed, so as to ensure the rationality and fairness of dynamic spectrum management.

[0101] According to an exemplary embodiment of the present disclosure, after each spectrum is allocated, the number of times the spectrum is allocated can be accumulated and added by one, that is, the occupancy status of the spectrum can be updated in time based on the occupancy of the spectrum, so as to provide a reference for the subsequent allocation of spectrum to new users, thereby ensuring the rationality of subsequent spectrum allocation.

[0102] According to an exemplary embodiment of the present disclosure, when each occupied spectrum is released, the number of times the spectrum is allocated can be reduced by one, that is, the occupancy status of the spectrum can be updated in time based on the occupancy of the spectrum, so as to provide a reference for subsequent allocation of spectrum to new users, thereby ensuring the rationality of subsequent spectrum allocation.

[0103] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0104] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A spectrum access method, characterized in that: include: Obtaining a maximum reward value corresponding to the current user to be accessed, wherein the maximum reward value is a maximum value among a plurality of reward values ​​corresponding to a plurality of spectrums one by one, and a reward value corresponding to each spectrum in the plurality of spectrums is a reward value obtained by assuming that the spectrum is allocated to the current user to be accessed; Determine whether to allow the spectrum to be allocated to the current user to be accessed based on the maximum reward value and / or the total number of users who have currently accessed the multiple spectrums; In response to determining that it is allowed to allocate spectrum to the current user to be connected, a spectrum corresponding to the maximum reward value among the multiple spectrums is allocated to the current user to be connected as a target spectrum.

2. The spectrum access method according to claim 1, characterized in that: The determining whether to allow the spectrum to be allocated to the current user to be accessed includes: Calculate the difference between the maximum reward value corresponding to the current user to be connected and the maximum reward value corresponding to the previous user who has connected; In response to the difference being greater than or equal to a preset threshold, it is determined that spectrum allocation is allowed for the current user to access.

3. The spectrum access method according to claim 1, characterized in that: The determining whether to allow the spectrum to be allocated to the current user to be accessed includes: estimating the total number of users who have currently accessed the multiple spectrums when spectrum is allocated to the current waiting user; comparing the estimated total quantity with a preset quantity threshold; In response to the estimated total number being less than or equal to the preset number threshold, it is determined to allow spectrum to be allocated to the current user to be connected.

4. The spectrum access method according to claim 1, characterized in that: The spectrum access method further includes: In response to allocating the target spectrum to the current user to be accessed, the number of times the target spectrum is allocated is increased by one, wherein the number of times the target spectrum is allocated is used to calculate a reward value corresponding to the target spectrum.

5. The spectrum access method according to claim 4, characterized in that: The spectrum access method further includes: In response to a user currently occupying the target spectrum releasing occupation of the target spectrum, the number of times the target spectrum is allocated is reduced by one.

6. A spectrum access device, characterized in that: include: A reward value acquisition module is configured to acquire a maximum reward value corresponding to a current user to be connected, wherein the maximum reward value is a maximum value among a plurality of reward values ​​corresponding to a plurality of spectrums one by one, and a reward value corresponding to each spectrum in the plurality of spectrums is a reward value obtained by assuming that the spectrum is allocated to the current user to be connected; A spectrum determination module is configured to determine whether to allow the spectrum to be allocated to the current user to be accessed based on the maximum reward value and / or the total number of users who have currently accessed the multiple spectrums; The allocation module is configured to allocate a spectrum corresponding to the maximum reward value among the multiple spectrums as a target spectrum to the current user to be connected in response to determining that the spectrum is allowed to be allocated to the current user to be connected.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the spectrum access method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the spectrum access method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the spectrum access method according to any one of claims 1 to 5 is implemented.

Images