A dynamic spectrum access method based on deep multi-user DRQN
By building a time series and combining the LSTM layer, the problem that the existing technology cannot effectively deal with large state space and partial observability is solved, and efficient dynamic spectrum access is achieved, reducing computing costs and improving spectrum access is achieved. success rate.
Patent Information
- Application Number
- CN202110956011.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-08-19
AI Technical Summary
Existing spectrum access methods cannot effectively adapt to complex real-world models, especially in the case of large state space and partial observability, and cannot effectively process the real state of the user, resulting in high computational costs.
Using a dynamic spectrum access method based on deep multi-user DRQN, the user's historical state value is stored by building a time series, and combining the network architecture of the LSTM layer and the fully connected layer, the DRQN network is trained to estimate the user's real state and realize dynamic spectrum connection. enter.
This method enables the DRQN network to adapt to complex real environments more effectively, reduce computing costs, and realize efficient dynamic spectrum access, significantly reduce the number of users' collisions and improve the success rate of spectrum access.
Smart Images

Figure CN113891327B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of spectrum access, and more specifically, to a dynamic spectrum access method based on deep multi-user DRQN. Background Art
[0002] With the rapid development of science and technology, various mobile communication devices have exploded, resulting in more and more mobile devices accessing the Internet. Therefore, the effective use of spectrum resources has become increasingly important, and the problem of spectrum scarcity has become increasingly serious. The growing demand for wireless communications and the scarcity of spectrum have triggered the development of efficient dynamic spectrum access solutions for emerging wireless network technologies. In 1999, Dr. Joseph Mitola introduced the concept of cognitive radio in his doctoral thesis. Since the adoption of the concept of cognitive radio, it has been highly valued by communication experts from various countries and has been continuously studied in depth. The adoption of cognitive radio provides a new direction and field for radio services. Its main task is to manage spectrum resources and provide high-quality services to users. In recent years, people have shown considerable interest in developing distributed optimization algorithms and learning algorithms to manage effective spectrum access between users. Distributed reinforcement learning is the application of reinforcement learning ideas and algorithms to distributed environments. The early distributed algorithm is to extend Q-learning to distributed systems. The full name of this improved algorithm is independent Q-learning (IQL). This algorithm is to execute the Q-learning algorithm independently for each user. Although this algorithm is suitable for distributed decision-making, its stability is not good. In recent years, some researchers have proposed the Deep Q-Network (DQN) based on Q-learning.
[0003] The good features of various DSA models designed for medium access control (MAC) can solve the problem of how to allocate channel usage rights when there is competition for the use of shared channels in local area networks. Deep reinforcement learning (DRL) (or deep Q-learning) has attracted a lot of attention in recent years because it can provide a good approximation of target values (called Q-values) when dealing with very large state and action spaces. In contrast to Q-learning methods that perform well in small-scale models but poorly in large-scale models, DRL combines deep neural networks with Q-learning, called deep Q-networks (DQNs), to overcome this problem. DQNs are used to map from states to actions in large-scale models in order to maximize Qvalues. In a recent Nature paper published by DeepMind, a DRL algorithm was developed that can teach computers how to play Atari games directly from screen pixels, and it showed strong performance in many test games. Soon after, scholars developed DRL algorithms for teaching multiple players how to communicate to maximize shared utility, and some players performed well in MNIST games and switch puzzles. In recent years, people have paid more and more attention to using DRL methods in various other fields. Due to the large state space in wireless connected devices and the partially observed nature of spectrum area management, it is hypothesized that incorporating DRL approaches into the design of DSA algorithms has great potential to provide efficient solutions for real-world complex spectrum access settings.
[0004] Existing spectrum access methods are generally not able to effectively adapt to process more complex real-world models. A dynamic spectrum access method based on a bidirectional long short-term memory network is proposed in Publication No. CN112672359A (publication date 2021-04-16). This scheme optimizes the DQN reinforcement learning algorithm by using a bidirectional long short-term memory network to form a DBRQN algorithm model, and inputs historical action data into the DBRQN algorithm model to predict dynamic spectrum holes, perceive the predicted dynamic spectrum holes, obtain the optimal channel access strategy, and then dynamically access idle spectrum holes to achieve dynamic spectrum sharing. However, this scheme does not construct a time series to predict the user's true state, and cannot enable the DQN network to better handle large state spaces and partial observability. Summary of the invention
[0005] In order to overcome the defects of the above-mentioned prior art, such as no time series is constructed to predict the real state of the user and the DQN network cannot better handle large state space and partial observability, the present invention provides a dynamic spectrum access method based on deep multi-user DRQN (Deep Recurrent Q-learning Network, DRQN).
[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0007] S1: Build a DRQN network, take channels as user-selectable actions, and select each channel randomly for users;
[0008] S2: Calculate the probability of each user accessing each channel, select actions based on the probability, and obtain the state value and reward value of each user based on the selected action;
[0009] S3: For one of the users, the next observation value of the user is obtained based on the state value and reward value of the user;
[0010] S4: Obtain the user's next state value and next reward value according to the selected action and the user's next observation value;
[0011] S5: Repeat steps S3 to S4, and store the action selected by the user and the state value, reward value, and observation value of each step as experience samples in the memory bank;
[0012] S6: Create a time series and store the state value obtained at each step in the time series;
[0013] S7: Execute steps S3 to S6 for all users to obtain a memory bank storing experience samples of all users and a time series storing state values of each step of all users;
[0014] S8: Extracting experience samples from the memory bank to perform centralized training on the DRQN network, completing centralized training of the DRQN network;
[0015] S9: The time series and the action selected by the user are input into the trained DRQN network, and the trained DRQN network completes the access to the dynamic spectrum.
[0016] Preferably, the DRQN network has a four-layer network architecture, including an input layer, an LSTM layer and two fully connected layers connected in sequence.
[0017] Preferably, in step S1, each user is allowed to transmit a data packet with a certain attempt probability in each transmission time slot of the channel; if there are n users and k channels, and it is assumed that each user has a data packet to transmit in a given time slot, there are two cases: n≤k and n>k; if n≤k, all users transmit and no collision occurs; if n>k, multiple users transmit in the same channel, collisions occur between users, and the channel is an action that can be randomly selected by the user.
[0018] Preferably, in step S2, after calculating the probability of each user accessing each channel, the channel with the highest probability of being selected is selected as the action selected by the user, and the state value and reward value of each user are obtained according to the action.
[0019] Preferably, in step S2, the specific method of calculating the probability of each user accessing each channel and making the selection is:
[0020] The simulated annealing algorithm is used to calculate the probability of each user accessing each channel. The Boltzmann distribution is used in the value function of the DRQN network. The probability of each channel is normalized by temperature. The calculation formula of the probability P(a) is:
[0021]
[0022] Where β represents temperature, α is a constant between 0 and 1, Q(a) is the value function, a represents the number of users, K represents an integer greater than or equal to 0, and K is less than n;
[0023] Then the channel with the highest probability is selected as the action selected by the user, and the state value and reward value of each user are obtained according to the action.
[0024] Preferably, in step S2, the user's action is specifically represented as:
[0025] a i =[0, 1, 2, ..., k]
[0026] a i =0 means the user accesses an idle channel, a i = k means the user selects channel k to access; where,
[0027] Preferably, in step S2, the state value is determined by a state space, and the state space includes the channel C selected by the user. n , remaining channel state A k and whether the channel is occupied successfully, where whether the channel is occupied successfully is represented by the reward function R(t);
[0028] At time t, the state set of i users is:
[0029] S t =[S 1 , S 2 , ..., S i ]
[0030] The status of the i-th user is:
[0031] S i =[C 1 , C2 , ..., C n , A 1 , A 2 , ..., A k ,R(t)]
[0032] Among them, there are:
[0033]
[0034]
[0035] Preferably, in step S2, the reward value is determined by a reward function, and it is defined that two or more users select the same channel as a channel collision, and it is defined that a user accesses a non-idle channel without a channel collision as a successful channel access. 0 The reward value is 1; when the user accesses an empty channel or accesses a non-idle channel and a channel collision occurs, the access channel fails, and I 1 Indicates that, and the reward value is 0;
[0036] Define R(t) as the reward function, and the calculation formula is:
[0037]
[0038] Preferably, in step S3, the observed value is determined by an observation function;
[0039] Observation is defined as the observation function. The specific formula is as follows:
[0040] observation=[O 1 , O 2 , ..., O n , A 1 , A 2 , ..., A k ]
[0041] On is the channel access status of user n:
[0042] O n =(C n , R(t)).
[0043] Preferably, in step S5, the memory bank is a bidirectional first-in-first-out queue with a capacity of 1000, which is used to store experience samples.
[0044] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0045] The present invention provides a dynamic spectrum access method based on deep multi-user DRQN. The time series constructed in the method can be used to store the historical state values of users and serve as the input of the trained DRQN network. The time series can enable the DRQN network to estimate the real state of the users, so that the users can more effectively adapt to the general complex real environment during the dynamic spectrum access process, while overcoming the expensive computing requirements caused by the large state space and partial observability of the problem. By adopting the trained DRQN network, a large number of users can finally complete the efficient dynamic spectrum access process. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is the algorithm step diagram;
[0047] Figure 2 It is an environmental interaction diagram between multiple users;
[0048] Figure 3 A network architecture diagram for multi-user spectrum access;
[0049] Figure 4 is the algorithm flow chart;
[0050] Figure 5 It is a multi-user spectrum access system model;
[0051] Figure 6 Comparison curve of loss function;
[0052] Figure 7 The cumulative number of collisions for all users;
[0053] Figure 8 Cumulative rewards for all users;
[0054] Fig. 9 is the success rate of multi-user spectrum access. DETAILED DESCRIPTION
[0055] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0056] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the drawings.
[0057] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0058] Example 1
[0059] A dynamic spectrum access method based on deep multi-user DRQN, such as Figure 1 As shown, the following steps are included:
[0060] S1: Build a DRQN network, take channels as user-selectable actions, and select each channel randomly for users;
[0061] The communication environment for user channel allocation in cognitive radio is modeled as a multi-agent environment, and a distributed algorithm is used to solve the problem of multi-user access to available channels. In the multi-user dynamic spectrum access problem, each user interacts with the unknown communication environment to gain experience, and then uses the experience to guide its own action strategy. Although resource sharing is a competitive game, in order to improve the overall performance of the system, that is, to minimize the cumulative number of collisions when users access and increase the number of successful user accesses, the same reward function is designed for each user, and strategies are shared between users to convert it into a fully cooperative game, such as Figure 2 As shown;
[0062] Specifically, the specific operations of constructing the DRQN network are as follows: The DRQN network has a four-layer network architecture, including an input layer, an LSTM layer, and two fully connected layers connected in sequence, such as Figure 3 As shown; among which:
[0063] Input layer: Input a vector of size 2K+2 into the DRQN network. The first K+1 input vector represents the channel selected by the user in the previous time slot, that is, the user's action. Specifically, when the user selects channel K, the channel is set to 1, and the remaining k channels are set to 0. If the user transmits the data packet successfully, the last element of the vector is set to 1. If the user data packet transmission fails or is not executed, the last element of the vector is set to 0.
[0064] LSTM layer: Because the network state is partially observable for each user and is dynamically non-Markovian, the classic DQN performs poorly in this situation. Based on this, an LSTM layer is added to DQN to maintain the internal state and the observations accumulated over time, which allows the network to estimate the true state based on past experience;
[0065] Fully connected layers: Both fully connected layers consist of 128 neurons each, and the last fully connected layer outputs the Q value of the value function.
[0066] Specifically, n users and k channels are set. In each time slot, each user is allowed to transmit a data packet with a certain trial probability. In a given time slot, it is assumed that each user has a data packet to transmit, where there are two cases: n≤k and n>k. If n≤k, all users transmit and no collision occurs. If n>k, multiple users transmit on the same channel and collisions occur between users. n and k satisfy the following formula: n={1, 2, ...n}, k={1, 2, ...k}. If only one user transmits on a certain channel in a given time slot, the transmission of channel k is successful and the user successfully accesses channel k. If multiple users transmit on the same channel, collisions occur between users, such as Figure 4 shown.
[0067] S2: Calculate the probability of each user accessing each channel, select actions based on the probability, and obtain the state value and reward value of each user based on the selected action;
[0068] This step first requires a judgment process, the process is as follows: before the judgment, set a random sample value that obeys the "0-1" uniform distribution, the value range of the random sample is [0, 1), set the exploration rate for the DRQN network, and the set initial exploration rate is 0.02. If the exploration rate is greater than the random sample value, the centralized training process cannot be entered; if the exploration rate is less than the random sample value, the centralized training process begins;
[0069] This algorithm is divided into two stages, namely learning (training) and implementation (execution). Among them, the focus is on centralized training and distributed execution, such as Figure 5 As shown. In the centralized training part, in each time slot, each user randomly selects a number of samples (mini-batch) from the memory bank (the memory bank is also called experience replay, which is a bidirectional first-in-first-out queue with a capacity of 1000. The target parameters of Q(s, a) in DRQN are the same for each user. Centralized training can use the experience collected by all users to train a single DRQN network, thereby greatly reducing the memory and computing resources required for centralized training. Unlike the DQN network, the DRQN network needs to create an additional time series (history input), denoted as H. H is a bidirectional queue with a capacity of 5, which is first-in-first-out like experience replay.
[0070] Then it enters the distributed execution phase. The exploration rate of the DRQN network decays as the time slot passes until it reaches the minimum. Then it provides channels for users to access and selects actions in combination with the simulated annealing algorithm. Unlike DQN, the DRQN network does not need to t , a t ) uses the Boltzmann distribution, but in Q(H t , a t ) t is the input of the DRQN network in the previous step, i.e. H t =LSTM(H t-1 , a t ), then normalize the probability of each action with temperature (β), and then select the action with the maximum probability. The calculation formula of action probability P(a) is:
[0071]
[0072] Where β represents temperature, α is a constant between 0 and 1, Q(a) is the value function, a represents the number of users, K represents an integer greater than or equal to 0, and K is less than n;
[0073] In this step, adding the simulated annealing algorithm can avoid local optimality and make the network develop towards the global optimality. After every 50 time slots, β decays. If the current β value is β 1 , then the β in the next state 2 =β 1 -0.001.
[0074] The user's actions are specifically expressed as:
[0075] a i =[0, 1, 2, ..., k]
[0076] a i =0 means the user accesses an idle channel, a i = k means that the user chooses channel k to access; all users have the same action space,
[0077] The state value is determined by the state space, which includes the channel C selected by the user. n , remaining channel state A k and whether the channel is occupied successfully, where whether the channel is occupied successfully is represented by the reward function R(t);
[0078] At time t, the state set of i users is:
[0079] S t =[S 1 , S 2, ..., S i ]
[0080] The status of the i-th user is:
[0081] S i =[C 1 , C 2 , ..., C n , A 1 , A 2 , ..., A k ,R(t)]
[0082] Among them, there are:
[0083]
[0084]
[0085] The reward value is determined by the reward function. It is defined that two or more users select the same channel as a channel collision, and it is defined that a user accesses a non-idle channel without a channel collision as a successful access to the channel. 0 The reward value is 1; when the user accesses an empty channel or accesses a non-idle channel and a channel collision occurs, the access channel fails, and I 1 Indicates that, and the reward value is 0;
[0086] Define R(t) as the reward function, and the calculation formula is:
[0087]
[0088] S3: For one of the users, the next observation value of the user is obtained based on the state value and reward value of the user;
[0089] The observation value is determined by the observation function. Observation is defined as the observation function. The specific formula is as follows:
[0090] observation=[O 1 , O 2 , ..., O n , A 1 , A 2 , ..., A k ]
[0091] On is the channel access status of user n:
[0092] O n =(C n , R(t)).
[0093] S4: Obtain the user's next state value and next reward value according to the selected action and the user's next observation value;
[0094] S5: Repeat steps S3 to S4, and store the action selected by the user and the state value, reward value, and observation value of each step as experience samples in the memory bank;
[0095] S6: Create a time series and store the state value obtained at each step in the time series;
[0096] S7: Execute steps S3 to S6 for all users to obtain a memory bank storing experience samples of all users and a time series storing state values of each step of all users;
[0097] S8: Extracting experience samples from the memory bank to perform centralized training on the DRQN network, completing centralized training of the DRQN network;
[0098] S9: The time series and the action selected by the user are input into the trained DRQN network, and the trained DRQN network completes the access to the dynamic spectrum.
[0099] After the training is completed, the time series and the actions selected by the user are input into the input layer of the DRQN network to carry out the dynamic spectrum access process for multiple users.
[0100] Example 2
[0101] Testing of algorithm effectiveness:
[0102] At the same time, the collaborative distributed RANDOM, DQN, and DRQN were tested. The number of time slots for centralized training was set to 50,000, the number of users was set to 3, and the accessible channels were set to 2. The goal was to use the algorithm to minimize access collisions among the three users, increase rewards, and improve the success rate of multi-user spectrum access and channel utilization. Figure 6 This is a comparison curve of the loss functions of the two algorithms. Figure 6 As can be seen from the figure, both algorithms can achieve good convergence effects after iterative learning. The loss function of the DQN-based algorithm converges to the minimum faster than the DRQN-based algorithm. This is because the DQN-based algorithm disrupts the correlation, thereby greatly speeding up the convergence of the loss function, while the long short-term memory needs to be connected with past experience, so the loss function will not converge to the minimum immediately.
[0103] Figure 7 and Figure 8 The comparison of the cumulative number of collisions and the comparison of the cumulative rewards using two different neural networks are described respectively. The results show that the DRQN algorithm can significantly reduce the number of collisions for users. Figure 7 and Figure 8It can be seen that the algorithm based on DRQN converges around the 20,000th time slot and converges to around 32,000; the algorithm based on DQN converges around the 35,000th time slot and finally converges to around 60,000. Although the number of collisions of the RANDOM algorithm during the test is less than that of DQN, RANDOM will not converge, so it is not difficult to imagine that in the following period of time, RANDOM will be higher than the other two algorithms. Figure 8 In the figure, compared with the DQN-based algorithm, the cumulative reward of the DRQN-based algorithm is significantly higher, both in terms of the rate of change of reward increase and the reward value.
[0104] Fig. 9 The success rate comparison curve of the two algorithms is depicted. It is defined that if one user successfully accesses the channel, it is considered a success. 1,000 time slots are extracted each time for statistics, and 50 times are extracted continuously, that is, 50,000 time slot rounds. From the simulation results, it can be seen that whether it is stability or success rate, DRQN is obviously better than DQN and RANDOM as a whole. The success rate of the strategy learned by users through DRQN in spectrum access finally converges to 1, that is, in each time slot, there must be a channel successfully accessed by the user, which greatly improves the utilization rate of the channel.
[0105] DRQN's algorithm performs well in the problem of dynamic spectrum access, especially in reducing loss values, algorithm convergence, reducing the number of collisions, increasing cumulative rewards, and improving the success rate of spectrum access. This also highlights that the algorithm has better learning ability and environmental adaptability, better stability, better algorithm performance, and improves the network performance of the overall allocation system.
[0106] For previous reinforcement learning algorithms such as Q-learning and DQN, users cannot access the complete network status information online, so it is often difficult to calculate the best strategy. Although optimal structured solutions have been developed for some specific situations, most existing research focuses on designing spectrum access protocols for specific models to provide effective but non-optimal structured solutions.
[0107] The present invention improves the ordinary single-agent DQN so that it can adapt to the multi-user channel access algorithm and match its complex multi-household channel allocation environment, and finally complete the distributed implementation and decision-making. In order to solve the two shortcomings in DQN, this method proposes a deep recurrent neural network (Deep Recurrent Q-learning Network, DRQN) for partially observable Markov processes, and replaces the fully connected layer in the original DQN with an LSTM network structure. In the actual environment, users rarely obtain relatively complete state information, so the Markov property is lost. The partially observable Markov decision process (POMDP) can well express this dynamic characteristic that cannot be fully obtained.
[0108] Based on DQN, this method combines the characteristics of recurrent neural networks and combines LSTM with DQN. The LSTM introduced by RNN can better handle partially observable situations, and DRQN can better approximate the actual Q function and learn better strategies. Finally, the simulated annealing algorithm is added to the distributed execution stage of the algorithm to avoid local optimality and make the network develop towards the global optimality. In summary, this method has a significant improvement in channel allocation.
[0109] The terms used to describe the positional relationship in the accompanying drawings are only used for illustrative purposes and should not be construed as limiting this patent; it is clear that the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to list all the implementation methods here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A dynamic spectrum access method based on deep multi-user DRQN, characterized in that: The following steps are involved: S1: Build a DRQN network, take channels as user-selectable actions, and select each channel randomly for users; S2: Calculate the probability of each user accessing each channel, select actions based on the probability, and obtain the state value and reward value of each user based on the selected action; S3: For one of the users, the next observation value of the user is obtained based on the state value and reward value of the user; S4: Obtain the user's next state value and next reward value according to the selected action and the user's next observation value; S5: Repeat steps S3 to S4, and store the action selected by the user and the state value, reward value, and observation value of each step as experience samples in the memory bank; S6: Create a time series and store the state value obtained at each step in the time series; S7: Execute steps S3 to S6 for all users to obtain a memory bank storing experience samples of all users and a time series storing state values of each step of all users; S8: Extracting experience samples from the memory bank to perform centralized training on the DRQN network, completing centralized training of the DRQN network; S9: input the time series and the action selected by the user into the trained DRQN network, and the trained DRQN network completes the access of the dynamic spectrum; In step S1, in each transmission time slot of the channel, each user is allowed to transmit a data packet with a certain trial probability; if there are n users and k channels, and each user has a data packet to transmit in a given time slot, there are two cases: n≤k and n>k; if n≤k, all users transmit and no collision occurs; if n>k, multiple users transmit in the same channel, collisions occur between users, and the channel is an action that can be randomly selected by the user; In step S2, the user's action is specifically represented as: a i =[0,1,2,...,k] a i =0 means the user accesses an idle channel, a i = k means the user selects channel k to access; where, In step S2, the state value is determined by the state space, which includes the channel C selected by the user. n , remaining channel state A k and whether the channel is occupied successfully, where whether the channel is occupied successfully is represented by the reward function R(t); At time t, the state set of i users is: S t =[S1,S2,...,S i ] The status of the i-th user is: S i =[C1,C2,...,C n ,A1,A2,...,A k ,R(t)] Among them, there are: In step S2, the reward value is determined by a reward function, and it is defined that two or more users select the same channel as a channel collision, and it is defined that a user accesses a non-idle channel and no channel collision occurs as a successful channel access, which is represented by I0, and the reward value is 1; when a user accesses an empty channel or accesses a non-idle channel and a channel collision occurs, it is defined as a channel access failure, which is represented by I1, and the reward value is 0; Define R(t) as the reward function, and the calculation formula is: In step S3, the observed value is determined by the observation function; Observation is defined as the observation function. The specific formula is as follows: observation=[O1,O2,...,O n ,A1,A2,...,A k ] O n Access channel status for user n: O n =(C n ,R(t))。 2. The method for dynamic spectrum access based on deep multi-user DRQN according to claim 1, characterized in that: In step S1, the DRQN network has a four-layer network architecture, including an input layer, an LSTM layer and two fully connected layers connected in sequence.
3. The dynamic spectrum access method based on deep multi-user DRQN according to claim 1, characterized in that: In step S2, after calculating the probability of each user accessing each channel, the channel with the highest probability of being selected is selected as the action selected by the user, and the state value and reward value of each user are obtained according to the action.
4. The method for dynamic spectrum access based on deep multi-user DRQN according to claim 1, characterized in that: In step S5, the memory bank is a bidirectional first-in-first-out queue with a capacity of 1000, which is used to store experience samples.
