Credible electromagnetic spectrum management and control method based on block chain and deep reinforcement learning

Abstract

The embodiment of the invention relates to the technical field of trusted electromagnetic spectrum management and control, in particular to a trusted electromagnetic spectrum management and control method based on a block chain and deep reinforcement learning, and the method comprises the steps that each secondary user obtains spectrum idle information through spectrum sensing, and a main user carries out the right weighted voting, a plurality of secondary users are selected as block outlet nodes to form a new block, and spectrum coins are distributed to each secondary user in the new block; each secondary user uses the spectrum currency to auct the plurality of idle frequency bands provided by the primary user, and an auction strategy is quickly made in the limited spectrum currency and the plurality of specific idle frequency bands through an intelligent agent based on deep reinforcement learning training; and each secondary user participates in the auction based on the made auction strategy, the intelligent contract deployed on the block chain processes the auction, deducts the spectrum coins of each secondary user who succeeds in the auction, returns the spectrum coins of each secondary user who fails in the auction, and extracts a part of the spectrum coins obtained by the auction as the reward of each node for operating the intelligent contract.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of trusted electromagnetic spectrum management and control, and in particular to a trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning. Background Art

[0002] Spectrum refers to the range of electromagnetic waves that widely exist in electromagnetic space. It is the basis for controlling and allocating electromagnetic resources. Effective spectrum management is crucial to achieving more reasonable and efficient use of electromagnetic space. In recent years, efficient spectrum management has become a hot topic of research. Cognitive Radio (CR) technology can achieve efficient spectrum management by dynamically allocating spectrum holes to secondary users. Li et al. considered the cooperation between nodes in a dynamic spectrum environment and proposed a dynamic spectrum access (DSA) scheme based on federated deep reinforcement learning to improve spectrum and access efficiency. Liu et al. perceived massive spectrum data through collaborative large spectrum data sensing, and optimized spectrum selection strategy based on reinforcement learning to enable cognitive vehicle networks to select the best spectrum for communication based on the results of spectrum data sensing. In addition, Netalkar et al. proposed a new DSA algorithm using the IEEE 1900.5.2 spectrum consumption model, which is suitable for spectrum deconfliction in large-scale wireless network environments. Kaur et al. studied the DSA paradigm for multi-user wireless network with imperfect feedback, with the goal of maximizing network utility.

[0003] However, spectrum abuse is a common problem in shared spectrum access systems. Abuse in DSA systems will degrade the quality of communication services and even prevent secondary users from using DSA. Spectrum regulators usually handle these abuses manually, which is neither efficient nor practical. With the increasing frequency of spectrum sharing in future technologies, manual investigation of such violations is impossible. In addition, dynamic spectrum access and sharing methods do not support a real-time spectrum trading market.

[0004] In the context of the rapid development of 5G and 6G, the relevance of blockchain technology to dynamic spectrum management is multifaceted. Blockchain technology provides a powerful solution to protect sensitive information by emphasizing enhanced security and decentralized data management. These technologies simplify the supply chain and dynamic spectrum resource management through transparent and tamper-proof data exchange, while the integration of smart contracts promotes automated protocols and improves efficiency. Cheng et al. studied a blockchain-based spectrum trading mechanism for citizen broadband radio services, proposed a queuing mechanism for coexistence group transactions, and a new parameter called "network characteristics" for prioritizing spectrum transactions in different queues. Fernand et al. proposed an energy-saving consensus mechanism for blockchain spectrum management, called "distributed proof of significance", which is used to enable DSA and detect spectrum violations. Weiss et al. explored the application of blockchain in radio spectrum management, fully considered the general advantages and limitations of blockchain solutions, and studied the potential application of blockchain solutions in four categories of spectrum sharing. Abognah et al. provided a blockchain-based model for dynamic spectrum sharing, which represents spectrum access rights as tokenized assets and allows these spectrum tokens to be traded between multiple users on a distributed ledger using smart contracts. Xue et al. proposed a new decentralized distributed spectrum trading protocol based on blockchain, which can effectively prevent distributed denial of service attacks, and verified its security against mainstream attacks such as double-consumption attacks through simulation analysis.

[0005] Through the above analysis, the inventors of this application found that the currently proposed spectrum management technology has the following defects:

[0006] 1) In the traditional static spectrum allocation method, the primary users who hold the right to use a specific spectrum range rarely use certain frequency bands, which leads to the generation of spectrum holes. Traditional static spectrum management and control cannot solve this defect, resulting in a waste of spectrum resources.

[0007] 2) Abuse in the currently proposed dynamic spectrum access methods will degrade the quality of communication services and even prevent secondary users from using dynamic spectrum access. Spectrum regulators usually handle these abuses manually, which is neither efficient nor practical. With the increasing frequency of spectrum sharing in future technologies, manual investigation of such violations will be impossible.

[0008] 3) The proposed blockchain-based dynamic spectrum management methods seem to be efficient and robust because of the decentralization, tamper-proof and transparency characteristics of blockchain. However, considering the cost, processing and energy limitations, the use of traditional consensus mechanisms in blockchain networks is too expensive. Summary of the invention

[0009] In view of this, the embodiments of the present application propose a trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning. By introducing blockchain technology and deep reinforcement learning technology, the problems of inefficient spectrum management and control and high cost of traditional blockchain consensus mechanism are solved, and the safe, efficient and dynamic allocation of spectrum resources is achieved, the spectrum utilization rate is improved, and the shortage of spectrum resources is alleviated.

[0010] To achieve the above-mentioned purpose, the embodiment of the present application proposes a trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning, the method comprising: each secondary user obtains spectrum idle information through spectrum sensing, the primary user performs equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of obtaining spectrum idle information, selects several secondary users as block nodes, forms new blocks based on each block node, and distributes spectrum coins to each secondary user in the new block; wherein the primary user is a legitimate user, the seller of the spectrum transaction, and a communication device authorized or occupied within a specific spectrum range, the secondary user is a communication device that wants to use the idle frequency band of the primary user, and the secondary user has no exclusive right to use the primary user's spectrum. The spectrum usage rights occupied by the main user; each sub-user in the new block uses spectrum coins to auction several idle frequency bands provided by the main user, and quickly makes auction strategies based on limited spectrum coins and specific idle frequency bands through intelligent agents trained based on deep reinforcement learning; each sub-user in the new block participates in the auction based on the auction strategy made, and the auction is processed by the smart contract deployed on the blockchain, and the bids of each user participating in the auction are stored in the smart contract, the auction results are run, the spectrum coins of each user who succeeds in the auction are deducted, the spectrum coins of each user who fails in the auction are returned, and a portion of the spectrum coins obtained from the auction is extracted according to a preset ratio as the reward for each node running the smart contract.

[0011] To achieve the above objectives, the embodiments of the present application also propose a trusted electromagnetic spectrum management and control system based on blockchain and deep reinforcement learning, including: a consensus proof module, an intelligent decision-making module and a spectrum trading module; the consensus proof module is used to instruct each secondary user to obtain spectrum idle information through spectrum sensing, instruct the primary user to perform equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of obtaining spectrum idle information, select several secondary users as block production nodes, form new blocks based on each block production node, and distribute spectrum coins to each secondary user in the new block, wherein the primary user is a legitimate user, the seller of the spectrum transaction, and a communication device authorized or occupied within a specific spectrum range, the secondary user is a communication device that wants to use the idle frequency band of the primary user, and the secondary user is a communication device that wants to use the idle frequency band of the primary user. Users do not have exclusive spectrum usage rights; the intelligent decision-making module is used to instruct each sub-user in the new block to use spectrum coins to auction several idle frequency bands provided by the main user, and quickly make auction strategies among limited spectrum coins and specific idle frequency bands through intelligent agents trained based on deep reinforcement learning; the spectrum trading module is used to instruct each sub-user in the new block to participate in the auction based on the auction strategy made, and the auction is processed by the smart contract deployed on the blockchain. The bids of each user participating in the auction are stored in the smart contract, the auction results are run, the spectrum coins of each user who succeeds in the auction are deducted, the spectrum coins of each user who fails in the auction are returned, and a portion of the spectrum coins obtained from the auction is extracted according to a preset ratio as the reward for each node running the smart contract.

[0012] To achieve the above objectives, an embodiment of the present application also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning.

[0013] To achieve the above objectives, an embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the above-mentioned trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning.

[0014] The trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning proposed in the embodiments of the present application provides a new solution for the allocation of 6G communication and Web3.0 spectrum resources, and proposes a new energy-saving consensus mechanism, called "distributed hierarchical proof", which encourages secondary users in the blockchain to perform spectrum sensing, thereby collecting full-spectrum sensing data and detecting spectrum violations. By introducing blockchain technology and deep reinforcement learning technology, the problems of inefficient spectrum management and control and high cost of traditional blockchain consensus mechanisms are solved, and the safe, efficient and dynamic allocation of spectrum resources is achieved, the spectrum utilization rate is improved, and the shortage of spectrum resources is alleviated.

[0015] Optionally, in the initial stage, the primary user creates a genesis block and determines the initial rights and interests based on the spectrum resources at his disposal; in the blockchain network, each secondary user obtains spectrum idle information through spectrum sensing, and the primary user performs equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of obtaining spectrum idle information, and selects several secondary users as block-producing nodes, and the weight of the equity-weighted voting is proportional to the size of the initial rights and interests determined by the primary user; each block-producing node selects a preset number of transactions from the transaction pool to be processed for verification, packages them into a new block, combines the selected transactions with the hash value of the previous block and the metadata of other block headers, calculates the hash value of the block header of the new block through a hash function, and broadcasts the generated new block to other nodes in the blockchain network; after receiving the new block broadcast, other nodes in the blockchain network verify the validity of the new block, including verifying the correctness of the transaction and the legitimacy of the signature.

[0016] Optionally, the distributing spectrum coins to each secondary user in the new block includes: distributing spectrum coins proportional to the size of the spectrum idle information obtained by each secondary user in the new block as rewards; wherein, each 1MB of spectrum idle information obtained corresponds to 1 spectrum coin reward.

[0017] Optionally, each secondary user in the new block uses spectrum coins to auction several idle frequency bands provided by the primary user, and an intelligent agent trained based on deep reinforcement learning quickly makes an auction strategy among limited spectrum coins and several specific idle frequency bands, including:

[0018] Setting Environment Sets for:

[0019]

[0020] Among them, {Ch 1 ,Ch 2 ,Ch 3 ,…,Ch k ,…,Ch n} represents all n frequency bands of the primary user, Ch k =0 means that the kth frequency band is not idle and does not participate in the auction. k =1 means the kth frequency band is idle and can participate in the auction, Sp i Indicates the current number of spectrum coins of the i-th agent, Nb i Indicates the urgency of the communication task of the i-th agent;

[0021] Set Action Set It is expressed as:

[0022]

[0023] Among them, Ac k represents the agent's bid for the kth frequency band;

[0024] The reward R is obtained according to the following conditions:

[0025]

[0026] Among them, st c It represents the step length consumed by the agent after completing the set communication task, that is, the total number of time slots consumed, st max Indicates the maximum step size set for the communication task, st min represents the step length that can achieve the best communication service quality, r max Indicates the maximum number of rewards set;

[0027] Initialize a random policy π θ , and use it to collect N trajectories. The collected N trajectories are expressed as:

[0028]

[0029] Among them, τ i represents the i-th trajectory, represents the environmental state at the tth step in the ith trajectory, represents the action taken by the agent at step t in the i-th trajectory, represents the immediate reward for the action taken by the agent at step t;

[0030] Based on the collected N trajectories, we get the trajectory benefit G(τ i ), for G(τ i ) is normalized, and the normalized return of the i-th trajectory is expressed as:

[0031]

[0032]

[0033]

[0034] Among them, R i is the profit of the ith trajectory, is the normalized representation of the return of the ith trajectory;

[0035] The return of the trajectory includes the first part and Part II In action a t It has already appeared before, for action a t No contribution, In action a t After the action a t Have contribution;

[0036] Strategy π θ After one update, it becomes π θ′ , the strategy update process is forcibly restricted by limiting the ratio of the new strategy to the old strategy, which can be expressed as:

[0037] clip[ρ t (θ′),1-∈,1+∈]

[0038]

[0039] Among them, ∈ represents the constraint value, clip(·) represents the clipping function;

[0040] Based on the restrictions of the strategy update process, the objective function is obtained. According to the objective function, the agent is continuously trained and updated, so that the agent trained based on deep reinforcement learning can achieve the goal of auctioning spectrum in a timely manner. The objective function is expressed as:

[0041]

[0042] Among them, E t (·) indicates taking the expectation, L CLIP (θ ′ ) represents the objective function.

[0043] Optionally, the smart contract is deployed on the blockchain and jointly verified and executed by each node in the blockchain network. The code and execution results of the smart contract are permanently recorded on the blockchain and cannot be tampered with or deleted. The code and execution logic of the smart contract are publicly visible. Each node supports viewing, verifying and auditing the code and execution logic of the smart contract. The smart contract is written in a specific programming language, including but not limited to Solidity and Vyper. After the auction results are run out, the auction information of this auction is stored in the block. The data structure of each block is a Merkel tree, which is a binary tree data structure used to verify and ensure the integrity of large amounts of data.

[0044] Optionally, the Merkel tree constructs a tree structure by dividing data into blocks of fixed size and calculating hash values ​​step by step, including: construction of leaf nodes, the data set is divided into blocks of fixed size, a hash value is calculated for each block, and each calculated hash value is used as a leaf node of the Merkle Tree; hash calculation and node construction, leaf nodes are hashed in pairs, and the calculation result is used as the value of the parent node. If there are an odd number of leaf nodes, the last leaf node will be copied once to ensure pairing, and the process of hash calculation and node construction is repeated step by step until only one root node is left; verification of the root node, the final root node is the root of the Merkle Tree, and the root hash value of the root node is used as proof of data integrity; wherein, the hash value obtained by verification calculation is compared with the known root hash value to verify the integrity of the data.

[0045] Optionally, the primary user is a communication system with high priority and protection requirements, including but not limited to government agency communication systems, military communication systems and licensed telecommunications operators, and the primary user has priority access rights; the secondary user is an ordinary mobile user device, including but not limited to smart phones, laptops, wireless sensor networks and Internet of Things devices, and the secondary user has a certain cognitive ability, that is, a certain spectrum perception ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of a trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning provided in one embodiment of the present application;

[0047] Figure 2 It is a structural diagram of a trusted electromagnetic spectrum management and control system based on blockchain and deep reinforcement learning provided in another embodiment of the present application;

[0048] Figure 3 is a schematic diagram of the results of a simulation experiment with 5 channels provided in another embodiment of the present application;

[0049] Figure 4 is a schematic diagram of the results of a simulation experiment with 10 channels provided in another embodiment of the present application;

[0050] Figure 5 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0052] In order to solve the problem of high cost and poor control effect of traditional spectrum control methods, an embodiment of the present application proposes a trusted electromagnetic spectrum control method based on blockchain and deep reinforcement learning, which is applied to electronic devices. This embodiment and the following embodiments are all described by taking the blockchain system as an example. The implementation details of the trusted electromagnetic spectrum control method based on blockchain and deep reinforcement learning proposed in this embodiment are described in detail below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0053] The specific process of the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0054] Step 101: Each secondary user obtains spectrum idle information through spectrum sensing. The primary user performs equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of obtaining spectrum idle information, selects several secondary users as block generation nodes, forms new blocks based on each block generation node, and distributes spectrum coins to each secondary user in the new block.

[0055] In the specific implementation, the primary user is a legitimate user, the seller of spectrum transactions, and a communication device that is authorized or occupied within a specific spectrum range. The secondary user is a communication device that wants to use the idle frequency band of the primary user, and the secondary user does not have exclusive spectrum use rights. Each secondary user obtains spectrum idle information through spectrum sensing. The primary user performs equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of obtaining spectrum idle information, selects several secondary users as block nodes, forms new blocks based on each block node, and distributes spectrum coins to each secondary user in the new block. Among them, the more authorized frequency bands the primary user has, the greater the equity.

[0056] In one example, the primary user is a communication system with high priority and protection requirements, including but not limited to government agency communication systems, military communication systems, and licensed telecommunications operators. The primary user enjoys the primary right to use the spectrum, has a higher priority and priority access rights, and in the blockchain system, the primary user will sell the idle frequency band in the form of several time slots when the licensed frequency band is not needed.

[0057] In one example, secondary users refer to other wireless devices other than primary users, which do not have exclusive spectrum usage rights. Secondary users are generally ordinary mobile user devices, including but not limited to smartphones, laptops, wireless sensor networks and IoT devices. Secondary users have certain cognitive abilities, that is, they have certain spectrum perception abilities and can analyze the spectrum usage of primary users through spectrum perception so as to transmit without interference in unused spectrum. Secondary users auction the idle frequency bands sold by primary users to obtain the right to use them.

[0058] In one example, in the initial stage, the primary user creates a genesis block and determines the initial rights and interests based on the spectrum resources controlled by the primary user. In the blockchain network, each secondary user obtains spectrum idle information through spectrum sensing. The primary user performs equity-weighted voting based on the speed of spectrum sensing and the accuracy of spectrum idle information obtained by each secondary user. The blockchain system selects several secondary users as block-producing nodes. The weight of the primary user's equity-weighted voting is proportional to the size of the initial rights and interests determined by the primary user. Each selected secondary user, that is, each block-producing node, selects a preset number of transactions from the transaction pool to be processed for verification and packages them into a new block. The selected transactions are combined with the hash value of the previous block and other block header metadata (such as timestamp, difficulty target, etc.). Each block-producing node calculates the hash value of the block header of the new block through a hash function and broadcasts the generated new block to other nodes in the blockchain network. After receiving the new block broadcast, other nodes in the blockchain network verify the validity of the new block, including verifying the correctness of the transaction and the legitimacy of the signature. In this way, new blocks are continuously generated and broadcasted.

[0059] It is worth noting that the process of calculating the hash value of the block header of a new block is similar to the process of miners packaging transactions in the traditional blockchain consensus proof-of-work mechanism, but does not require a lot of computing work.

[0060] In one example, when the main user (or the blockchain system) distributes spectrum coins to each secondary user in a new block, it distributes spectrum coins proportional to the size of the spectrum idle information obtained by each secondary user in the new block as rewards; wherein, each 1MB of spectrum idle information obtained corresponds to 1 spectrum coin reward.

[0061] In step 102, each secondary user in the new block uses spectrum coins to auction several idle frequency bands provided by the primary user, and uses an intelligent agent trained based on deep reinforcement learning to quickly make auction strategies among limited spectrum coins and several specific idle frequency bands.

[0062] In the specific implementation, each secondary user in the new block uses spectrum coins to auction several idle frequency bands provided by the primary user. The intelligent agent trained based on deep reinforcement learning quickly makes an auction strategy among the limited spectrum coins and several specific idle frequency bands. In fact, the intelligent agent is in state S t Next, perform action A t , when the environment state is transferred to S t+1 After that, you will get an instant reward R t+1 , and the goal of deep reinforcement learning is to find a strategy π to maximize the expectation of the cumulative reward and .

[0063] In one example, setting the environment set for:

[0064]

[0065] Among them, {Ch 1 ,Ch 2 ,Ch 3 ,…,Ch k ,…,Ch n} represents all n frequency bands of the primary user, Ch k =0 means that the kth frequency band is not idle and does not participate in the auction. k =1 means the kth frequency band is idle and can participate in the auction, Sp i Indicates the current number of spectrum coins of the i-th agent, Nb i Indicates the urgency of the communication task of the i-th agent;

[0066] Set Action Set It is expressed as:

[0067]

[0068] Among them, Ac k represents the agent's bid for the kth frequency band;

[0069] The reward R is obtained according to the following conditions:

[0070]

[0071] Among them, st c It represents the step length consumed by the agent after completing the set communication task, that is, the total number of time slots consumed, st max Indicates the maximum step size set for the communication task, st min represents the step length that can achieve the best communication service quality, r max Indicates the maximum number of rewards set.

[0072] In one example, when training, the agent first initializes a random policy π θ , and use it to collect N trajectories. The collected N trajectories are expressed as:

[0073]

[0074] Among them, τ i represents the i-th trajectory, represents the environmental state at the tth step in the ith trajectory, represents the action taken by the agent at step t in the i-th trajectory, represents the immediate reward for the action taken by the agent in step t. A sequence of several such sets of environment-action-reward pairs is the trajectory until the maximum step length is reached or the communication task is completed;

[0075] Therefore, the profit of the trajectory G(τ i ), for G(τ i ) is normalized, and the normalized return of the i-th trajectory is expressed as:

[0076]

[0077] Among them, R i is the profit of the ith trajectory, is the normalized representation of the return of the ith trajectory;

[0078] The return of the trajectory includes the first part and Part II In action a t It has already appeared before, for action a t No contribution, In action at After the action a t Have contribution;

[0079] Strategy π θ After one update, it becomes π θ′ , the strategy update process is forcibly restricted by limiting the ratio of the new strategy to the old strategy, which can be expressed as:

[0080] clip[ρ t (θ′),1-∈,1+∈]

[0081]

[0082] Among them, ∈ represents the constraint value, clip(·) represents the clipping function;

[0083] Based on the restrictions of the policy update process, the objective function is obtained, which is expressed as:

[0084]

[0085] Among them, E t (·) indicates taking the expectation, L CLIP (θ′) represents the objective function;

[0086] By continuously training and updating the intelligent agent according to the objective function, the intelligent agent trained based on deep reinforcement learning can achieve the goal of being able to auction the spectrum in a timely manner. This can solve the problem that the traditional auction process goes through multiple rounds of auctions, which is time-consuming and energy-consuming. The trained intelligent agent only needs to go through one auction, which greatly saves time and improves spectrum utilization.

[0087] Step 103: Each user in the new block participates in the auction based on the auction strategy. The auction is processed by a smart contract deployed on the blockchain. The bids of each user participating in the auction are stored in the smart contract. The auction results are run, and the spectrum coins of each user whose auction succeeds are deducted. The spectrum coins of each user whose auction fails are returned, and a portion of the spectrum coins obtained from the auction is extracted according to a preset ratio as a reward for each node running the smart contract.

[0088] In the specific implementation, each user in the new block participates in the auction based on the auction strategy made, and the auction is handled by the smart contract deployed on the blockchain. The blockchain system stores the bids of each user participating in the auction into the smart contract, runs the auction results, deducts the spectrum coins of each user who succeeds in the auction, returns the spectrum coins of each user who fails in the auction, and extracts a portion of the spectrum coins obtained from the auction according to a preset ratio as the reward for each node running the smart contract.

[0089] A smart contract is an automatically executed contract written in the form of computer code, designed to execute and implement the terms of the contract on a blockchain system or distributed ledger technology. A smart contract can automatically execute and monitor the performance of the contract without a third party. The code of a smart contract is designed to execute automatically without human intervention. Once the preset conditions are met, the contract will automatically trigger execution according to the pre-written logic.

[0090] Smart contracts are deployed on the blockchain and are jointly verified and executed by each node in the blockchain network, which means that there is no single centralized entity controlling the execution of the contract, and all participants can verify the results of the contract. The code and execution results of the smart contract are permanently recorded on the blockchain and cannot be tampered with or deleted, which ensures the transparency and traceability of the contract. The code and execution logic of the smart contract are publicly visible, and each node supports viewing, verification and auditing the code and execution logic of the smart contract, which provides higher transparency, and due to the automatic execution of the contract and the security mechanism of the blockchain, the smart contract has a higher security. Smart contracts are written in specific programming languages, including but not limited to Solidity and Vyper. After the smart contract is deployed, it is interacted with and executed by calling the contract's functions. The execution results and status of the smart contract will be recorded on the blockchain and can be verified and audited at any time.

[0091] In one example, after the blockchain system runs the auction results, it needs to store the auction information of this auction in the block. The data structure of each block is Merkel tree, which is a binary tree data structure used to verify and ensure the integrity of large amounts of data.

[0092] In one example, the Merkel tree constructs a tree structure by dividing data into fixed-size blocks and calculating hash values ​​step by step, including three steps: leaf node construction, hash calculation and node construction, and root node verification.

[0093] To construct leaf nodes, the data set is divided into blocks of fixed size, a hash value is calculated for each block, and each calculated hash value is used as a leaf node of the Merkle Tree.

[0094] Hash calculation and node construction: leaf nodes are hashed in pairs, and the calculation results are used as the value of the parent node. If there are an odd number of leaf nodes, the last leaf node will be copied once to ensure pairing. The process of hash calculation and node construction is repeated upward step by step until only one root node is left.

[0095] Verification of the root node. The final root node is the root of the Merkle Tree. The root hash value of the root node is used as proof of data integrity. The hash value calculated by verification is compared with the known root hash value to verify the integrity of the data.

[0096] Merkle Tree can efficiently verify the legitimacy and integrity of large amounts of data because it only needs to calculate a small number of hash values ​​to verify the entire data set.

[0097] This embodiment provides a new solution for the allocation of spectrum resources for 6G communications and Web3.0, and proposes a new energy-saving consensus mechanism, called "distributed hierarchical proof", which incentivizes secondary users in the blockchain to perform spectrum sensing, thereby collecting full-spectrum sensing data and detecting spectrum violations. By introducing blockchain technology and deep reinforcement learning technology, the problems of inefficient spectrum management and control and high cost of traditional blockchain consensus mechanisms are solved, and the safe, efficient and dynamic allocation of spectrum resources is achieved, the spectrum utilization rate is improved, and the shortage of spectrum resources is alleviated.

[0098] Correspondingly, another embodiment of the present application proposes a trusted electromagnetic spectrum management and control system based on blockchain and deep reinforcement learning. The details of the trusted electromagnetic spectrum management and control system based on blockchain and deep reinforcement learning proposed in this embodiment are described in detail below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this example.

[0099] Figure 2 It is a structural diagram of the trusted electromagnetic spectrum management and control system based on blockchain and deep reinforcement learning proposed in this embodiment, including: a consensus proof module 201, an intelligent decision-making module 202 and a spectrum trading module 203.

[0100] The consensus proof module 201 is used to instruct each secondary user to obtain spectrum idle information through spectrum sensing, instruct the primary user to perform equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of obtaining spectrum idle information, select several secondary users as block production nodes, form new blocks based on each block production node, and distribute spectrum coins to each secondary user in the new block, wherein the primary user is a legitimate user, the seller of the spectrum transaction, and a communication device authorized or occupied within a specific spectrum range, and the secondary user is a communication device that wants to use the idle frequency band of the primary user, and the secondary user does not have exclusive spectrum use rights.

[0101] The intelligent decision-making module 202 is used to instruct each secondary user in the new block to use spectrum coins to auction several idle frequency bands provided by the primary user, and to quickly make auction strategies based on limited spectrum coins and several specific idle frequency bands through an intelligent agent trained based on deep reinforcement learning.

[0102] The spectrum trading module 203 is used to instruct each user in the new block to participate in the auction based on the auction strategy. The auction is processed by a smart contract deployed on the blockchain. The bids of each user participating in the auction are stored in the smart contract, the auction results are run, the spectrum coins of each user who succeeds in the auction are deducted, the spectrum coins of each user who fails in the auction are returned, and a portion of the spectrum coins obtained from the auction is extracted according to a preset ratio as the reward for each node running the smart contract.

[0103] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0104] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0105] In another example, in order to evaluate the performance of the trusted electromagnetic spectrum management method based on blockchain and deep reinforcement learning proposed in this application, the technicians conducted a simulation experiment. In the simulation experiment, a trusted electromagnetic spectrum management system based on blockchain and deep reinforcement learning is considered. Figure 3 and Figure 4 is a comparison of the time taken for a round of auction between the proposed timely auction mechanism and the traditional auction. The vertical axis is the number of secondary users in the environment, and the horizontal axis is the time taken for a round of auction. Figure 3 The number of channels is 5. Figure 4 The number of channels is 10. Obviously, when there are more channels, the auction takes longer. Here, each user can bid for all idle channels at the same time. This is also the case when the number of users is the same. Figure 3 Compare Figure 4 It can be seen that when there are 5 channels, the time taken for a timely auction increases from 0.13s to 0.14s as the number of users increases, while the time taken for a regular auction increases from 0.17s to 0.27s as the number of users increases; when there are 10 channels, the time taken for a timely auction increases from 0.17s to 0.20s as the number of users increases, while the time taken for a regular auction increases from 0.35s to 0.50s as the number of users increases. Not only does the regular auction take longer when the number of users is the same, but its time taken increases with the number of users to a greater extent than the timely auction.

[0106] Another embodiment of the present application provides an electronic device, the structure of which is as follows: Figure 5 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning in the above-mentioned embodiments.

[0107] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0108] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0109] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning described in the above method embodiment.

[0110] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0111] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning, It is characterized in that include: Each secondary user obtains spectrum idle information through spectrum sensing. The primary user performs equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of spectrum idle information obtained, selects several secondary users as block generation nodes, forms new blocks based on each block generation node, and distributes spectrum coins to each secondary user in the new block; the primary user is a legitimate user, the seller of spectrum transactions, and a communication device authorized or occupied within a specific spectrum range. The secondary user is a communication device that wants to use the idle frequency band of the primary user, and the secondary user does not have exclusive spectrum use rights; Each secondary user in the new block uses spectrum coins to auction several idle frequency bands provided by the primary user. Through the intelligent agent trained based on deep reinforcement learning, it can quickly make auction strategies among limited spectrum coins and several specific idle frequency bands. Each user in the new block participates in the auction based on the auction strategy made. The auction is processed by the smart contract deployed on the blockchain. The bids of each user participating in the auction are stored in the smart contract, the auction results are run, the spectrum coins of each user who succeeds in the auction are deducted, the spectrum coins of each user who fails in the auction are returned, and a portion of the spectrum coins obtained from the auction is extracted according to the preset ratio as the reward for each node running the smart contract.

2. According to the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning according to claim 1, It is characterized in that In the initial stage, the main user creates the genesis block and determines the initial rights and interests based on the spectrum resources he controls; In the blockchain network, each secondary user obtains spectrum idle information through spectrum sensing. The primary user performs equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of spectrum idle information obtained, and selects several secondary users as block nodes. The weight of the equity-weighted voting is proportional to the size of the initial equity determined by the primary user. Each block-producing node selects a preset number of transactions from the pending transaction pool for verification, packages them into a new block, combines the selected transactions with the hash value of the previous block and metadata of other block headers, calculates the hash value of the block header of the new block through a hash function, and broadcasts the generated new block to other nodes in the blockchain network; After receiving the new block broadcast, other nodes in the blockchain network verify the validity of the new block, including verifying the correctness of the transaction and the legitimacy of the signature.

3. According to the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning according to claim 2, It is characterized in that The distribution of spectrum coins to each secondary user in the new block includes: According to the size of spectrum idle information obtained by each user in the new block, spectrum coins proportional to the size of the spectrum idle information obtained are distributed to each user in the new block as rewards; Among them, every 1MB of spectrum idle information obtained corresponds to 1 spectrum coin reward.

4. According to the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning according to claim 1, It is characterized in that Each secondary user in the new block uses spectrum coins to auction several idle frequency bands provided by the primary user, and the intelligent agent trained based on deep reinforcement learning quickly makes auction strategies in limited spectrum coins and several specific idle frequency bands, including: Setting Environment Sets for: Among them, {Ch 1 ,Ch 2 ,Ch 3 ,…,Ch k ,…,Ch n } represents all n frequency bands of the primary user, Ch k =0 means that the kth frequency band is not idle and does not participate in the auction. k =1 means the kth frequency band is idle and can participate in the auction, Sp i Indicates the current number of spectrum coins of the i-th agent, Nb i Indicates the urgency of the communication task of the i-th agent; Set Action Set It is expressed as: Among them, Ac k represents the agent's bid for the kth frequency band; The reward R is obtained according to the following conditions: Among them, st c It represents the step length consumed by the agent after completing the set communication task, that is, the total number of time slots consumed, st max Indicates the maximum step size set for the communication task, st min represents the step length that can achieve the best communication service quality, r max Indicates the maximum number of rewards set; Initialize a random policy π θ , and use it to collect N trajectories. The collected N trajectories are expressed as: Among them, τ i represents the i-th trajectory, represents the environmental state at the tth step in the ith trajectory, represents the action taken by the agent at step t in the i-th trajectory, represents the immediate reward for the action taken by the agent at step t; Based on the collected N trajectories, we get the trajectory benefit G(τ i ), for G(τ i ) is normalized, and the normalized return of the i-th trajectory is expressed as: Among them, R i is the profit of the ith trajectory, is the normalized representation of the return of the ith trajectory; The return of the trajectory includes the first part and Part II In action a t It has already appeared before, for action a t No contribution, In action a t After the action a t Have contribution; Strategy π θ After one update, it becomes π θ′ , the strategy update process is forcibly restricted by limiting the ratio of the new strategy to the old strategy, which can be expressed as: clip[r t (η'),1-∈,1+∈] Among them, ∈ represents the constraint value, clip(·) represents the clipping function; Based on the restrictions of the strategy update process, the objective function is obtained. According to the objective function, the agent is continuously trained and updated, so that the agent trained based on deep reinforcement learning can achieve the goal of auctioning spectrum in a timely manner. The objective function is expressed as: Among them, E t (·) indicates taking the expectation, L CLIP (θ′) represents the objective function.

5. According to the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning according to claim 1, It is characterized in that Smart contracts are deployed on the blockchain and are jointly verified and executed by all nodes in the blockchain network. The code and execution results of smart contracts are permanently recorded on the blockchain and cannot be tampered with or deleted. The code and execution logic of smart contracts are publicly visible. All nodes support viewing, verification and auditing of the code and execution logic of smart contracts. Smart contracts are written in specific programming languages, including but not limited to Solidity and Vyper. After the auction results are run, the auction information of this auction is stored in the block. The data structure of each block is Merkel tree, which is a binary tree data structure used to verify and ensure the integrity of large amounts of data.

6. According to the trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning according to claim 5, It is characterized in that Merkel tree builds a tree structure by dividing data into fixed-size blocks and calculating hash values ​​level by level, including: The construction of leaf nodes, the data set is divided into blocks of fixed size, a hash value is calculated for each block, and each calculated hash value is used as the leaf node of the Merkle Tree; Hash calculation and node construction: leaf nodes are hashed in pairs, and the calculation results are used as the value of the parent node. If there are an odd number of leaf nodes, the last leaf node will be copied once to ensure pairing. The process of hash calculation and node construction is repeated upward step by step until only one root node is left; Verification of the root node. The final root node is the root of the Merkle Tree. The root hash value of the root node is used as proof of data integrity. The hash value calculated by verification is compared with the known root hash value to verify the integrity of the data.

7. The trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning according to any one of claims 1 to 6, It is characterized in that Primary users are communication systems with high priority and protection requirements, including but not limited to government agency communication systems, military communication systems and licensed telecommunications operators. Primary users have priority access rights; Secondary users are ordinary mobile user devices, including but not limited to smart phones, laptops, wireless sensor networks and Internet of Things devices. Secondary users have certain cognitive abilities, that is, they have certain spectrum perception abilities.

8. A trusted electromagnetic spectrum management and control system based on blockchain and deep reinforcement learning. It is characterized in that include: Consensus proof module, intelligent decision-making module and spectrum trading module; The consensus proof module is used to instruct each secondary user to obtain spectrum idle information through spectrum sensing, instruct the primary user to perform equity-weighted voting based on the speed of spectrum sensing of each secondary user and the accuracy of spectrum idle information obtained, select several secondary users as block generation nodes, form new blocks based on each block generation node, and distribute spectrum coins to each secondary user in the new block, wherein the primary user is a legitimate user, the seller of spectrum transactions, and a communication device authorized or occupied within a specific spectrum range, and the secondary user is a communication device that wants to use the idle frequency band of the primary user, and the secondary user does not have exclusive spectrum use rights; The intelligent decision-making module is used to instruct each secondary user in the new block to use spectrum coins to auction several idle frequency bands provided by the primary user. Through the intelligent agent trained based on deep reinforcement learning, it can quickly make auction strategies among limited spectrum coins and several specific idle frequency bands. The spectrum trading module is used to instruct each user in the new block to participate in the auction based on the auction strategy. The auction is handled by a smart contract deployed on the blockchain. The bids of each user participating in the auction are stored in the smart contract, the auction results are run, the spectrum coins of each user who succeeds in the auction are deducted, the spectrum coins of each user who fails in the auction are returned, and a portion of the spectrum coins obtained from the auction is extracted according to a preset ratio as the reward for each node running the smart contract.

9. An electronic device, It is characterized in that include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, it can implement a trusted electromagnetic spectrum management and control method based on blockchain and deep reinforcement learning as described in any one of claims 1 to 7.

Images