A spectrum access method and device based on quantum cycle reinforcement learning

Abstract

The application discloses a spectrum access method and device based on quantum cycle reinforcement learning, and relates to the technical fields of wireless communication and artificial intelligence. The method comprises the following steps: constructing a local history sequence by using a noise sensing result, a last time slot action and a confirmation feedback of each user, inputting the local history sequence into a cycle Q network with shared parameters, extracting time sequence features by using a cycle neural network, inputting the time sequence features into a parameterized quantum circuit after compression, obtaining a quantum feature vector by using an expected value of a Pauli-Z observable, obtaining Q values of each candidate action based on the quantum feature vector, training the cycle Q network with shared parameters by using a time difference target, outputting the trained cycle Q network with shared parameters, and outputting a dynamic spectrum access decision of each user in a current time slot based on the trained network. The application can improve normalized successful transmission throughput under limited training budget and strong sensing noise, and meanwhile, a multi-user distributed execution mode is reserved.

Description

Technical Field

[0001] This invention relates to the fields of wireless communication and artificial intelligence, and in particular to a spectrum access method and apparatus based on quantum cyclic reinforcement learning. Background Technology

[0002] Dynamic spectrum access is used to enable multiple secondary users to compete for a limited number of orthogonal channels without centralized coordination, thereby improving spectrum resource utilization. In practical wireless systems, secondary users typically cannot directly observe the actual busy / idle status of each channel, but can only obtain local sensing results with false alarms and missed detections, and receive confirmation feedback after the action is executed. Therefore, this type of access process simultaneously possesses characteristics of partial observability, multi-user coupling, and timing decision-making.

[0003] Existing dynamic spectrum access methods based on recurrent Q-learning can alleviate some observability problems by utilizing short-history observations, but the feature representation ability and training stability of pure classical models may be limited when perceptual noise is strong and training budget is limited. On the other hand, directly adopting global quantum policies or large-scale quantum structures will increase inference and training overhead and is prone to deviating from the actual input form of distributed local history decision-making.

[0004] Therefore, a dynamic access technology solution for imperfect spectrum awareness is needed, which can enable the model to introduce a compact quantum nonlinear feature enhancement mechanism while retaining the ability to extract local historical cyclic features, and improve access decision performance under limited budget conditions by adapting the training process to high-noise scenarios. Summary of the Invention

[0005] To address the technical challenges of insufficient historical observation representation, unstable training in noisy scenarios, and inadequate matching between quantum structures and local decision-making tasks in distributed multi-user dynamic spectrum access under imperfect spectrum sensing, this invention provides a spectrum access method and apparatus based on quantum cyclic reinforcement learning. The technical solution is as follows: On the one hand, a spectrum access method based on quantum cyclic reinforcement learning is provided, which is implemented by a spectrum access device based on quantum cyclic reinforcement learning. The method includes: S1. In a time-slotted dynamic spectrum access system containing N secondary users and K orthogonal channels, obtain the noise spectrum perception results of each user for each channel in each time slot, the access action of the previous time slot, and the confirmation feedback of the previous time slot. S2. Based on the noise spectrum perception results, the access action of the previous time slot, and the confirmation feedback of the previous time slot, construct a single time slot local observation, and construct a user local historical sequence from L consecutive single time slot local observations. S3. Input the local historical sequence into a recurrent Q-network with shared parameters, and obtain historical time-series features through recurrent neural network encoding; perform linear mapping on the historical time-series features to obtain classical features; S4. Compress the historical time series features into a low-dimensional classical input vector and input it into a parameterized quantum circuit. Measure the expected value of a preset Pauli-Z observable to obtain a quantum feature vector. S5. The classical features and quantum feature vectors are concatenated and fused to obtain the Q value of each candidate action; wherein, the candidate actions include: the silent action and the access action of selecting any channel; S6. Construct a time difference target, train a recurrent Q-network with shared parameters based on the Q-values ​​of each candidate action, and output the trained recurrent Q-network with shared parameters. S7. Input each user's local historical sequence into a recurrent Q-network based on the trained shared parameters in the current time slot, and output the dynamic spectrum access decision for each user in the current time slot.

[0006] On the other hand, a spectrum access device based on quantum cyclic reinforcement learning is provided, which is applied to a spectrum access method based on quantum cyclic reinforcement learning. The device includes: The sensing input module is used to acquire the noise spectrum sensing results of each user for each channel in each time slot, the access action of the previous time slot, and the confirmation feedback of the previous time slot in a time-slotted dynamic spectrum access system containing N secondary users and K orthogonal channels. The history construction module is used to construct a single-time-slot local observation based on the noise spectrum perception results, the access action of the previous time slot, and the confirmation feedback of the previous time slot, and to construct a user local history sequence from L consecutive single-time-slot local observations. The recurrent feature extraction module is used to input the local historical sequence into a recurrent Q-network with shared parameters, and obtain historical time-series features through recurrent neural network encoding; and to perform linear mapping on the historical time-series features to obtain classical features. The quantum feature generation module is used to compress the historical time series features into a low-dimensional classical input vector and input it into a parameterized quantum circuit to measure the expected value of a preset Pauli-Z observable to obtain a quantum feature vector. The feature fusion and action value output module is used to concatenate and fuse the classical features and quantum feature vectors to obtain the Q value of each candidate action; wherein, the candidate actions include: a silent action and an access action that selects any channel; The training module is used to construct the temporal difference objective, train the recurrent Q-network with shared parameters based on the Q-value of each candidate action, and output the trained recurrent Q-network with shared parameters. The decision execution module is used to input each user's local historical sequence into a recurrent Q-network based on the trained shared parameters in the current time slot, and output the dynamic spectrum access decision for each user in the current time slot.

[0007] On the other hand, a spectrum access device based on quantum cyclic reinforcement learning is provided, the spectrum access device based on quantum cyclic reinforcement learning comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the above-described spectrum access methods based on quantum cyclic reinforcement learning is implemented.

[0008] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of spectrum access based on quantum cyclic reinforcement learning.

[0009] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention employs local historical sequences and shared-parameter cyclic Q-networks, enabling multiple users to make distributed access decisions under partially observable conditions. It utilizes compact parameterized quantum circuits as nonlinear feature enhancement branches to improve expressive power in noisy scenarios without replacing the cyclic temporal backbone. Data re-upload enhances quantum feature expressive power without increasing the number of qubits. Course transfer training improves model initialization quality and training stability in high-perception-noise scenarios. Diagnostic metrics such as success rate and Jain index can be used to check for significant user starvation after deployment. This invention addresses the problems of insufficient historical observation representation, unstable training in noisy scenarios, and insufficient matching between quantum structures and local decision-making tasks in distributed multi-user dynamic spectrum access under imperfect spectrum perception. It also improves normalized successful transmission throughput under limited training budgets and strong perception-noise conditions while preserving the multi-user distributed execution mode. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a spectrum access method based on quantum cyclic reinforcement learning provided in an embodiment of the present invention; Figure 2This is a schematic diagram of a spectrum access system structure based on quantum cyclic reinforcement learning provided in an embodiment of the present invention; Figure 3 This is a comparison chart of network training convergence performance in a high-noise scenario provided by an embodiment of the present invention; Figure 4 This is a block diagram of a spectrum access device based on quantum cyclic reinforcement learning provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a spectrum access device based on quantum cyclic reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0013] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0014] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0015] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0016] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0017] This invention provides a spectrum access method based on quantum cyclic reinforcement learning. This method can be implemented by a spectrum access device based on quantum cyclic reinforcement learning, which can be a terminal or a server. Figure 1 The flowchart shown is for a spectrum access method based on quantum recurrent reinforcement learning. The processing flow of this method may include the following steps: S1. In a time-slotted dynamic spectrum access system containing N secondary users and K orthogonal channels, obtain the noise spectrum perception results of each user for each channel in each time slot, the access action of the previous time slot, and the confirmation feedback of the previous time slot.

[0018] This invention addresses a distributed multi-user dynamic spectrum access scenario under imperfect spectrum awareness conditions. For example... Figure 2 The diagram shown is a schematic diagram of a spectrum access system based on quantum recurrent reinforcement learning provided by an embodiment of the present invention.

[0019] In one feasible implementation, the present invention constructs a time-slotted distributed dynamic spectrum access system; wherein the system includes: N secondary users and K orthogonal channels, and each user chooses to remain silent or select one channel for access in time slot t.

[0020] In a time-slotted distributed dynamic spectrum access system, let N be the number of secondary users and K be the number of orthogonal channels. The action of user i in time slot t is represented as... Its action space is represented by the following formula (1): (1) Where 0 represents silence, and k>0 represents attempting to access the k-th channel; the actual busy / idle state of channel k in time slot t is denoted as , This indicates that the account is occupied by the main user. Indicates idle. The perception result of user i on channel k is denoted as... , This indicates that the perception is idle.

[0021] Optionally, the noise spectrum sensing results are used to characterize whether the channel is perceived as idle or busy, and the noise spectrum sensing results are constructed by the false alarm probability and the missed detection probability.

[0022] The noise spectrum perception result is expressed by the following formula (2): (2) in, Indicates the probability of a false alarm; This represents the probability of missed detection; where, successful user transmission requires simultaneously satisfying the following conditions: selecting a non-silent action, the selected channel being truly idle, and no other secondary users selecting the same channel. Its success indicator is expressed by the following formula (3): (3) in, This indicates a successful transmission by user i in time slot t. This indicates the channel selected by user i in time slot t. The true state of being busy or idle; This represents the action of user j in time slot t.

[0023] S2. Based on the noise spectrum perception results, the access action of the previous time slot, and the confirmation feedback of the previous time slot, construct a single time slot local observation, and construct the user's local historical sequence from L consecutive single time slot local observations.

[0024] Optionally, single-slot local observation includes: one-hot coding of the action in the previous time slot, noise perception vectors of each channel in the current time slot, and confirmation feedback from the previous time slot; wherein, the dimension of single-slot local observation is jointly determined by the number of channels and the action space.

[0025] In one feasible implementation, the single-slot local observation of user i is represented by the following formula (4): (4) in, This represents a single-slot local observation by user i; This represents the one-hot encoding of the action in the previous time slot; This represents the noise perception vector of each channel in the current time slot. ; This indicates confirmation feedback from the previous time slot.

[0026] In one feasible implementation, the user's local history sequence constructed from L consecutive single-slot local observations can be represented by the following formula (5): (5) In this embodiment of the invention, K=2 and L=5, the noise perception vector of each channel in the current time slot is K=2-dimensional; the confirmation feedback of the previous time slot is 1-dimensional, so the local observation dimension of a single time slot is 3+2+1=6-dimensional, and the user's local historical sequence is a sequence input of L×6=5×6.

[0027] S3. Input the local historical sequence into a recurrent Q-network with shared parameters, and obtain the historical time series features through recurrent neural network encoding; perform linear mapping on the historical time series features to obtain classic features.

[0028] The recurrent neural network is a single-layer LSTM network, where the LSTM network outputs a historical time-series feature vector with a preset feature dimension.

[0029] The process of obtaining historical time series features through recurrent neural network encoding can be represented by the following formula (6): (6) in, This represents the historical time-series feature vector output by the recurrent neural network; Indicates the local historical sequence.

[0030] Among them, Classical enhanced features can be obtained through linear mapping, where the linear mapping process can be represented by the following formula (7): (7) in, Indicates characteristics of historical time sequence Classical features obtained through residual linear mapping; This represents the residual mapping function, which consists of a linear layer and a nonlinear activation function.

[0031] Optionally, the recurrent Q-network with shared parameters includes: a recurrent neural network, parameterized quantum circuitry, and a fused output head; Among them, the recurrent neural network is used to perform time-series encoding on the user's local historical sequence, extract the historical dependency between the noise spectrum perception result, the access action of the previous time slot, and the confirmation feedback of the previous time slot, and output historical time-series features. Among them, the parameterized quantum circuit is used to receive the low-dimensional classical input vector obtained by compressing historical time-series features, and generate quantum feature vectors through quantum coding layer and trainable entangled layer to enhance the nonlinear feature expression capability of the network. The fusion output head is used to concatenate and fuse classical and quantum feature vectors and map them to the Q-values ​​of each candidate action.

[0032] S4. Compress the historical time series features into a low-dimensional classical input vector and input it into the parameterized quantum circuit. Measure the expected value of the preset Pauli-Z observable to obtain the quantum feature vector.

[0033] The historical time series features are compressed into a low-dimensional classical input vector that matches the number of qubits through linear mapping and hyperbolic tangent function, and then encoded into a parameterized quantum circuit with the corresponding number of qubits.

[0034] In one feasible implementation, the process of compressing historical time-series features into a low-dimensional classical input vector can be represented by the following formula (8): (8) in, Represents a low-dimensional classical input vector; The first trainable parameter; This is the second trainable parameter; in this embodiment of the invention, If the input is a 4-dimensional vector, then the expected value of the preset Pauli-Z observable is measured in the 4-bit parameterized quantum circuit to obtain the quantum eigenvector, which is expressed by the following formula (9): (9) in, Represents quantum eigenvectors; For classic input x and quantum circuit parameters A jointly determined quantum state; Let be the Pauli-Z observable on the j-th qubit.

[0035] S5. The classical features and quantum feature vectors are concatenated and fused to obtain the Q value of each candidate action; among which, the candidate actions include: the silent action and the access action of selecting any channel.

[0036] In one feasible implementation, the process of concatenating quantum feature vectors and classical features and inputting them into a fusion output head to obtain the Q-values ​​of each candidate action is represented by the following formula (10): (10) in, Indicates fusion characteristics; ( () represents the Q-value vector output for all candidate actions; This means mapping the fused features to the action value space.

[0037] Optionally, the parameterized quantum circuit includes: a quantum coding layer and a trainable entangled layer; by calculating the expectation of the Pauli-Z observable of each qubit, a quantum eigenvector with the same dimension as the number of qubits is obtained.

[0038] Among them, the parameterized quantum circuit adopts a data re-upload structure, repeatedly injecting the coding amount obtained by scaling the low-dimensional classical input vector into multiple re-upload blocks.

[0039] In one feasible implementation, to enhance the expressive power of the compact quantum circuit, this embodiment of the invention employs a data re-upload structure. In the b-th re-upload block, the encoded amount obtained by scaling the input vector is re-injected into the quantum circuit, as expressed by the following formula (11): (11) in, Indicates the input encoding unit; This represents an entangled unit containing trainable parameters; This represents the scaling parameter for the b-th re-uploaded block; This represents element-wise multiplication; the data re-upload structure improves the nonlinear feature representation capability by repeatedly injecting the same low-dimensional input without increasing the number of qubits.

[0040] S6. Construct a time difference target, train a recurrent Q-network with shared parameters based on the Q-values ​​of each candidate action, and output the trained recurrent Q-network with shared parameters.

[0041] Optionally, the specific implementation process of S6 includes S61-S62: S61. Construct a time-difference objective using instant successful transmission rewards and discount factors; In one feasible implementation, the time difference objective is expressed by the following formula (12): (12) in, This represents the time difference objective in time slot t; This indicates that the reward was successfully transmitted instantly. Indicates the discount factor; t represents the maximum action value obtained; t represents the time slot.

[0042] S62. The mean square error between the time difference target and the Q value of each current candidate action is used as the loss function; the gradient of the loss function is calculated, and the parameters of the recurrent neural network, parameterized quantum circuit and fused output head are updated by gradient descent, and the recurrent Q network with shared parameters after training is output. The loss function is expressed by the following formula (13): (13) Where M represents the number of samples in the mini-batch; Indicates the time difference objective; ( ) represents the network's characteristics of historical states. and action output The Q value.

[0043] In one feasible implementation, embodiments of the present invention can perform course transfer training in high-knowledge noise application scenarios, the specific process of which includes: Construct a source noise training scenario, set the false alarm probability and the missed detection probability as the first noise level, and train a recurrent Q network with shared parameters under this noise level according to the constructed time difference objective and loss function to obtain the source scenario network parameters; The source scene network parameters are used as the initialization parameters for the target noise training scene; the false alarm probability and false negative probability of the target noise training scene are higher than the first noise level, while keeping the number of users, number of channels, action space, reward function and network structure unchanged. In the target noise training scenario, continue to collect the local historical sequence, action, reward and the next time slot local historical sequence of each user, recalculate the time difference target, and continue to update the parameters of the recurrent neural network, parameterized quantum circuit and fusion output head based on the samples in the target noise training scenario; When the preset number of training rounds is reached, the average reward per time slot tends to stabilize, or the loss function is lower than the preset threshold, the target noise training is stopped, and a recurrent Q-network with shared parameters suitable for high-perception noise scenarios is output.

[0044] Among them, such as Figure 3 The figure shown is a comparison of network training convergence performance in a high-noise scenario provided by an embodiment of the present invention; the training convergence performance of deep recurrent Q network, re-upload hybrid quantum network and course transfer re-upload hybrid quantum network are compared; wherein, under the same number of training iterations, the average reward per time slot of course transfer re-upload hybrid quantum network is greater than that of deep recurrent Q network and re-upload hybrid quantum network.

[0045] S7. Input each user's local historical sequence into a recurrent Q-network based on the trained shared parameters in the current time slot, and output the dynamic spectrum access decision for each user in the current time slot.

[0046] The successful access decision conditions are: the channel selected by the user is actually idle, and only one secondary user selects the channel in the same time slot; the network training objective is to maximize the long-term discount benefit of the sum of the number of successful transmissions of each user.

[0047] In one feasible implementation, a recurrent Q-network with shared parameters is deployed after training, and each user inputs their local historical sequence into the recurrent Q-network with shared parameters in the current time slot, and the action with the largest Q value is selected as the access decision.

[0048] Optionally, after step S7, which involves each user inputting their local historical sequence into a recurrent Q-network based on trained shared parameters in the current time slot and outputting the dynamic spectrum access decision for each user in the current time slot, the system further includes: Calculate the success rate of each user, the minimum success rate of each user, and the Jain fairness index to diagnose the fairness of dynamic spectrum access decisions for each user in the current time slot.

[0049] The Jain fairness index is used to check whether there is a significant imbalance in access among users; the Jain fairness index is expressed by the following formula (14): (14) in, The success rate of the i-th user is represented by N, which is not used as a hard constraint training target in this embodiment, but is used to evaluate the user balance of the access results during the deployment or testing phase; N represents the number of secondary users participating in dynamic spectrum access.

[0050] In one feasible approach, a time-slotted distributed dynamic spectrum access system includes three secondary users and two orthogonal channels. A basic observation mode is adopted, with a single time-slot observation as a 6-dimensional vector, a history length L=5, an LSTM hidden dimension of 64, a quantum branch using 4 qubits and 2 entangled layers, and a data re-upload block count of 3. During training, the original successful transmission reward is used with a discount factor of 0.9, and a shared parameter Q-network is used to train on the local historical samples of all users. For high-sensitivity noise scenarios, training is first performed in a scenario where both the false alarm probability and the missed detection probability are 0.2, and then the obtained parameters are transferred to a scenario where both the false alarm probability and the missed detection probability are 0.3 for further training. During deployment, each user independently inputs its own local historical sequence and outputs either a silent or channel access action.

[0051] This invention employs local historical sequences and shared-parameter cyclic Q-networks, enabling multiple users to make distributed access decisions under partially observable conditions. It utilizes compact parameterized quantum circuits as nonlinear feature enhancement branches to improve expressive power in noisy scenarios without replacing the cyclic temporal backbone. Data re-upload enhances quantum feature expressive power without increasing the number of qubits. Course transfer training improves model initialization quality and training stability in high-perception-noise scenarios. Diagnostic metrics such as success rate and Jain index can be used to check for significant user starvation after deployment. This invention addresses the problems of insufficient historical observation representation, unstable training in noisy scenarios, and insufficient matching between quantum structures and local decision-making tasks in distributed multi-user dynamic spectrum access under imperfect spectrum perception. It also improves normalized successful transmission throughput under limited training budgets and strong perception-noise conditions while preserving the multi-user distributed execution mode.

[0052] Figure 4 This is a block diagram of a spectrum access device based on quantum cyclic reinforcement learning, provided in an embodiment of the present invention. This device is used for a spectrum access method based on quantum cyclic reinforcement learning. (Refer to...) Figure 4 The device includes a perception input module 410, a history construction module 420, a cyclic feature extraction module 430, a quantum feature generation module 440, a feature fusion and action value output module 450, a training module 460, and a decision execution module 470. Among them: The sensing input module 410 is used to acquire the noise spectrum sensing results of each user for each channel in each time slot, the access action of the previous time slot, and the confirmation feedback of the previous time slot in a time-slotted dynamic spectrum access system containing N secondary users and K orthogonal channels. The history construction module 420 is used to construct a single-time-slot local observation based on the noise spectrum perception result, the access action of the previous time slot, and the confirmation feedback of the previous time slot, and to construct a user local history sequence from L consecutive single-time-slot local observations. The recurrent feature extraction module 430 is used to input the local historical sequence into a recurrent Q-network with shared parameters, and obtain historical time-series features through recurrent neural network encoding; and to perform linear mapping on the historical time-series features to obtain classical features. The quantum feature generation module 440 is used to compress the historical time series features into a low-dimensional classical input vector and input it into a parameterized quantum circuit to measure the expected value of a preset Pauli-Z observable to obtain a quantum feature vector. The feature fusion and action value output module 450 is used to concatenate and fuse the classical features and quantum feature vectors to obtain the Q value of each candidate action; wherein, the candidate actions include: a silent action and an access action that selects any channel; Training module 460 is used to construct a time difference target, train a recurrent Q-network with shared parameters based on the Q-value of each candidate action, and output the trained recurrent Q-network with shared parameters. The decision execution module 470 is used to input each user's local historical sequence into a recurrent Q network based on the trained shared parameters in the current time slot, and output the dynamic spectrum access decision for each user in the current time slot.

[0053] Optionally, the noise spectrum perception result is used to characterize whether the channel is perceived as idle or busy, and the noise spectrum perception result is constructed by the false alarm probability and the missed detection probability.

[0054] Optionally, the single-slot local observation includes: one-hot coding of the action in the previous time slot, noise perception vectors of each channel in the current time slot, and confirmation feedback from the previous time slot; wherein, the dimension of the single-slot local observation is jointly determined by the number of channels and the action space.

[0055] Optionally, the shared-parameter recurrent Q-network includes: a recurrent neural network, a parameterized quantum circuit, and a fused output head; Among them, the recurrent neural network is used to perform time-series encoding on the user's local historical sequence, extract the historical dependency between the noise spectrum perception result, the access action of the previous time slot, and the confirmation feedback of the previous time slot, and output historical time-series features. Among them, the parameterized quantum circuit is used to receive the low-dimensional classical input vector obtained by compressing historical time-series features, and generate quantum feature vectors through quantum coding layer and trainable entangled layer to enhance the nonlinear feature expression capability of the network. The fusion output head is used to concatenate and fuse classical and quantum feature vectors and map them to the Q-values ​​of each candidate action.

[0056] Optionally, the parameterized quantum circuit includes: a quantum coding layer and a trainable entangled layer; The parameterized quantum circuit employs a data re-upload structure, repeatedly injecting the encoded amount obtained by scaling the low-dimensional classical input vector into multiple re-upload blocks.

[0057] Optionally, the training module 460 is used for: A time-difference objective is constructed using instant successful transmission rewards and discount factors; The mean squared error between the time difference target and the Q value of each candidate action is used as the loss function; the gradient of the loss function is calculated, and the parameters of the recurrent neural network, parameterized quantum circuit and fused output head are updated by gradient descent, and the recurrent Q network with shared parameters after training is output. The loss function is expressed by the following formula (15): (15) in, The training loss of the recurrent Q-network with shared parameters is represented by M; M represents the number of mini-batch samples. Represents the time difference objective of the m-th sample; ( ) indicates in the parameter The lower network has historical state characteristics and actions The output Q value.

[0058] Optionally, after the step of inputting each user's local historical sequence into a recurrent Q-network based on trained shared parameters in the current time slot and outputting the dynamic spectrum access decision for each user in the current time slot, the method further includes: Calculate the success rate of each user, the minimum success rate of each user, and the Jain fairness index to diagnose the fairness of dynamic spectrum access decisions for each user in the current time slot.

[0059] This invention employs local historical sequences and shared-parameter cyclic Q-networks, enabling multiple users to make distributed access decisions under partially observable conditions. It utilizes compact parameterized quantum circuits as nonlinear feature enhancement branches to improve expressive power in noisy scenarios without replacing the cyclic temporal backbone. Data re-upload enhances quantum feature expressive power without increasing the number of qubits. Course transfer training improves model initialization quality and training stability in high-perception-noise scenarios. Diagnostic metrics such as success rate and Jain index can be used to check for significant user starvation after deployment. This invention addresses the problems of insufficient historical observation representation, unstable training in noisy scenarios, and insufficient matching between quantum structures and local decision-making tasks in distributed multi-user dynamic spectrum access under imperfect spectrum perception. It also improves normalized successful transmission throughput under limited training budgets and strong perception-noise conditions while preserving the multi-user distributed execution mode.

[0060] Figure 5 This is a schematic diagram of the structure of a spectrum access device based on quantum cyclic reinforcement learning provided in an embodiment of the present invention, as shown below. Figure 5 As shown, spectrum access devices based on quantum recurrent reinforcement learning can include the above-mentioned... Figure 4 The spectrum access device based on quantum recurrent reinforcement learning is shown. Optionally, the spectrum access device 510 based on quantum recurrent reinforcement learning may include a first processor 2001.

[0061] Optionally, the spectrum access device 510 based on quantum recurrent reinforcement learning may also include a memory 2002 and a transceiver 2003.

[0062] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0063] The following is combined with Figure 5 A detailed introduction to each component of the quantum recurrent reinforcement learning-based spectrum access device 510 is provided below: The first processor 2001 is the control center of the spectrum access device 510 based on quantum recurrent reinforcement learning. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0064] Optionally, the first processor 2001 can perform various functions of the quantum recurrent reinforcement learning-based spectrum access device 510 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0065] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0066] In a specific implementation, as one example, the spectrum access device 510 based on quantum cyclic reinforcement learning may also include multiple processors, for example... Figure 5 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0067] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0068] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the quantum cyclic reinforcement learning-based spectrum access device 510. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0069] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0070] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0071] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected via the interface circuit of the quantum recurrent reinforcement learning-based spectrum access device 510. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0072] It should be noted that, Figure 5 The structure of the quantum recurrent reinforcement learning-based spectrum access device 510 shown in the figure does not constitute a limitation on the router. Actual quantum recurrent reinforcement learning-based spectrum access devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0073] Furthermore, the technical effects of the spectrum access device 510 based on quantum cyclic reinforcement learning can be referred to the technical effects of the spectrum access method based on quantum cyclic reinforcement learning described in the above method embodiments, and will not be repeated here.

[0074] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0075] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0076] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0077] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0078] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0079] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A spectrum access method based on quantum cyclic reinforcement learning, characterized in that, The method includes: S1. In a time-slotted dynamic spectrum access system containing N secondary users and K orthogonal channels, obtain the noise spectrum perception results of each user for each channel in each time slot, the access action of the previous time slot, and the confirmation feedback of the previous time slot. S2. Based on the noise spectrum perception results, the access action of the previous time slot, and the confirmation feedback of the previous time slot, construct a single time slot local observation, and construct a user local historical sequence from L consecutive single time slot local observations. S3. Input the local historical sequence into a recurrent Q-network with shared parameters, and obtain historical time-series features through recurrent neural network encoding; perform linear mapping on the historical time-series features to obtain classical features; S4. Compress the historical time series features into a low-dimensional classical input vector and input it into a parameterized quantum circuit. Measure the expected value of a preset Pauli-Z observable to obtain a quantum feature vector. S5. The classical features and quantum feature vectors are concatenated and fused to obtain the Q value of each candidate action; wherein, the candidate actions include: the silent action and the access action of selecting any channel; S6. Construct a time difference target, train a recurrent Q-network with shared parameters based on the Q-values ​​of each candidate action, and output the trained recurrent Q-network with shared parameters. S7. Input each user's local historical sequence into a recurrent Q-network based on the trained shared parameters in the current time slot, and output the dynamic spectrum access decision for each user in the current time slot.

2. The spectrum access method based on quantum recurrent reinforcement learning according to claim 1, characterized in that, The noise spectrum perception result is used to characterize whether the channel is perceived as idle or busy, and the noise spectrum perception result is constructed by the false alarm probability and the missed detection probability.

3. The spectrum access method based on quantum recurrent reinforcement learning according to claim 1, characterized in that, The single-slot local observation includes: one-hot coding of the action in the previous time slot, noise perception vectors of each channel in the current time slot, and confirmation feedback from the previous time slot; wherein, the dimension of the single-slot local observation is determined by the number of channels and the action space.

4. The spectrum access method based on quantum recurrent reinforcement learning according to claim 1, characterized in that, The shared-parameter recurrent Q-network includes: a recurrent neural network, parameterized quantum circuitry, and a fused output head; Among them, the recurrent neural network is used to perform time-series encoding on the user's local historical sequence, extract the historical dependency between the noise spectrum perception result, the access action of the previous time slot, and the confirmation feedback of the previous time slot, and output historical time-series features. Among them, the parameterized quantum circuit is used to receive the low-dimensional classical input vector obtained by compressing historical time-series features, and generate quantum feature vectors through quantum coding layer and trainable entangled layer to enhance the nonlinear feature expression capability of the network. The fusion output head is used to concatenate and fuse classical and quantum feature vectors and map them to the Q-values ​​of each candidate action.

5. The spectrum access method based on quantum recurrent reinforcement learning according to claim 1, characterized in that, The parameterized quantum circuit includes: a quantum coding layer and a trainable entangled layer; The parameterized quantum circuit employs a data re-upload structure, repeatedly injecting the encoded amount obtained by scaling the low-dimensional classical input vector into multiple re-upload blocks.

6. The spectrum access method based on quantum recurrent reinforcement learning according to claim 1, characterized in that, The construction of the temporal difference objective in S6 involves training a recurrent Q-network with shared parameters based on the Q-values ​​of each candidate action, and outputting the trained recurrent Q-network with shared parameters, including: S61. Construct a time-difference objective using instant successful transmission rewards and discount factors; S62. The mean square error between the time difference target and the Q value of each current candidate action is used as the loss function; the gradient of the loss function is calculated, and the parameters of the recurrent neural network, parameterized quantum circuit and fused output head are updated by gradient descent, and the recurrent Q network with shared parameters after training is output. The loss function is expressed by the following formula (1): （1） in, The training loss of the recurrent Q-network with shared parameters is represented by M; M represents the number of mini-batch samples. Represents the time difference objective of the m-th sample; ( ) indicates in the parameter The lower network has historical state characteristics and actions The output Q value.

7. The spectrum access method based on quantum recurrent reinforcement learning according to claim 1, characterized in that, After step S7, which involves each user inputting their local historical sequence into a recurrent Q-network based on trained shared parameters in the current time slot and outputting the dynamic spectrum access decision for each user in the current time slot, the system further includes: Calculate the success rate of each user, the minimum success rate of each user, and the Jain fairness index to diagnose the fairness of dynamic spectrum access decisions for each user in the current time slot.

8. A spectrum access device based on quantum cyclic reinforcement learning, wherein the spectrum access device based on quantum cyclic reinforcement learning is used to implement the spectrum access method based on quantum cyclic reinforcement learning as described in any one of claims 1-7, characterized in that, The device includes: The sensing input module is used to acquire the noise spectrum sensing results of each user for each channel in each time slot, the access action of the previous time slot, and the confirmation feedback of the previous time slot in a time-slotted dynamic spectrum access system containing N secondary users and K orthogonal channels. The history construction module is used to construct a single-time-slot local observation based on the noise spectrum perception results, the access action of the previous time slot, and the confirmation feedback of the previous time slot, and to construct a user local history sequence from L consecutive single-time-slot local observations. The recurrent feature extraction module is used to input the local historical sequence into a recurrent Q-network with shared parameters, and obtain historical time-series features through recurrent neural network encoding; and to perform linear mapping on the historical time-series features to obtain classical features. The quantum feature generation module is used to compress the historical time series features into a low-dimensional classical input vector and input it into a parameterized quantum circuit to measure the expected value of a preset Pauli-Z observable to obtain a quantum feature vector. The feature fusion and action value output module is used to concatenate and fuse the classical features and quantum feature vectors to obtain the Q value of each candidate action; wherein, the candidate actions include: a silent action and an access action that selects any channel; The training module is used to construct the temporal difference objective, train the recurrent Q-network with shared parameters based on the Q-value of each candidate action, and output the trained recurrent Q-network with shared parameters. The decision execution module is used to input each user's local historical sequence into a recurrent Q-network based on the trained shared parameters in the current time slot, and output the dynamic spectrum access decision for each user in the current time slot.

9. A spectrum access device based on quantum cyclic reinforcement learning, characterized in that, The spectrum access device based on quantum recurrent reinforcement learning includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images