A method for modeling and distributed implementation of multi-channel access considering service priority

Abstract

The application provides a multi-channel access modeling and distributed implementation method considering service priority, a general buffer queue model considering service priority classification is used, a new network benefit maximization constraint optimization model is established by combining the characteristics that different priority services transmitted by the same wireless resource have different benefits to the whole network, the model is converted into a Multi-agent reinforcement learning decision process, a deep learning framework of multi-link distributed dynamic channel access based on the model is constructed, all Agents are centrally trained by using the optimization target of the established model as a reward, and a corresponding distributed dynamic channel access method is obtained, and the Agents trained by the deep learning framework are loaded into the link transmitters for distributed execution, so that the dynamic spectrum access decision with the global service priority balance effect can be realized by using only the local observation of each link.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of dynamic spectrum access of wireless network, and particularly relates to a modeling and distributed implementation method of multi-channel access considering service priority. BACKGROUND

[0002] Recently, the contradiction between the increasing demand of wireless communication and the increasingly scarce spectrum resources is increasingly prominent, and the dynamic spectrum access technology allowing users to flexibly switch on the available wireless channel is a key means to improve the utilization rate of scarce spectrum resources. For multi-link dynamic spectrum access, three goals need to be considered: selecting the best condition channel, collision avoidance and on-demand access. Goal one is mainly achieved through local information obtained by channel estimation and spectrum sensing of the link, while goals two and three have poor performance when relying only on local information for decision-making. Especially for goal three, each link accesses the channel according to its own service demand, which is easy to implement, but it is difficult for each link to cooperate with access to balance the service load without the assistance of global information. Although the central spectrum resource scheduling can achieve spectrum access that meets all three goals, the required coordination signal overhead is large, which excessively consumes the scarce spectrum resources. A distributed spectrum access that can achieve acceptable performance, uses local information as input for channel access decision-making, and has small signal overhead is a widely considered implementation form in dynamic spectrum access problems.

[0003] In the traditional competitive-based distributed dynamic spectrum access mode, each node selfishly occupies wireless resources without considering the service demand of other links and the benefit of the whole network, and its performance deteriorates sharply with the increase of network load. Therefore, under the condition of no information interaction, it is a key challenge for dynamic spectrum access to achieve wireless resource allocation with global awareness. Considering the memory and generalization ability of neural networks in deep reinforcement learning, the present application regards each link as an Agent, proposes a novel modeling method of multi-channel access optimization problem based on the buffer queue model considering service priority, and uses the optimization goal of the problem as the reward to train all Agents centrally, thereby giving each link the ability to balance the global service priority only according to local information when distributedly executed. SUMMARY

[0004] In order to solve the problems of low utilization efficiency of wireless resources and lack of global service priority balance awareness in the traditional dynamic spectrum access method, the application proposes a modeling method of multi-channel access optimization problem considering service priority, and designs a distributed implementation method based on Multi-agent deep reinforcement learning, that is, the optimization objective of the above optimization problem model is taken as the reward to train all Agents centrally, and the trained Agents are loaded onto each link device for distributed execution, so that each link can make dynamic channel access decisions with global service priority balance capability only relying on local information.

[0005] The application proposes a modeling and distributed implementation method of multi-channel access considering service priority, including the following steps:

[0006] Step 1: establishing a multi-link dynamic channel access constraint optimization model considering service priority;

[0007] Step 2: converting the constraint optimization model established in step 1 into a Multi-agent reinforcement learning decision process, and constructing a deep learning framework for multi-link dynamic channel decision based on the constraint optimization model.

[0008] Step 3: taking the optimization objective of the constraint optimization model established in step 1 as the reward, centrally training all Agents of the deep learning framework constructed in step 2, loading the trained Agents onto the corresponding transmitters of each link in the network for distributed execution, inputting the local observations of each link in the current time slot, and thus obtaining the channel selected by the transmitter of each link in the current time slot for access.

[0009] The constraint optimization model in step 1 is:

[0010]

[0011]

[0012]

[0013] In the above model, N wireless links simultaneously access a wireless spectrum containing K orthogonal channels, the transmitter of each link has a service buffer queue containing L priorities, and the smaller the buffer queue label, the higher the priority, and the maximum buffer capacity of each queue is B max data packets. Each channel is divided into equal period time slots in time, and F consecutive time slots define a time frame. When all links have completed time synchronization, each time slot on each channel is called a transmission opportunity, and the capacity of all transmission opportunities is 1 data packet. In time slot t, the transmitter of the nth link makes a channel selection decision according to its own buffer queue ​Thus the feedback of each channel is obtained denotes the i-th channel is selected by the transmitter of the n-th link, otherwise not selected, denotes the information of the n-th link is successfully transmitted on the i-th channel, otherwise not successfully or no feedback due to not selecting the link.

[0014] ρ n (t) is the link transmission value variable:

[0015]

[0016] where B u is a constant for normalizing the link transmission value, is the weight of the l * priority queue, and has denotes the index of the highest priority queue in the priority queues which are not zero for the transmitter of the n-th link.

[0017] The benefit of the whole network generated by transmitting different link traffics on each transmission opportunity is:

[0018]

[0019] Optimization objective Opt1 denotes the benefit sum of all links in a time frame, and optimization objective Opt2 is the time slot index when all links complete transmission in the current time frame. Constraint condition C1 denotes the definition of the decision action of each link transmitter; constraint condition C2 denotes the update of the data packets in each level of the traffic buffer queue of each link transmitter; constraint condition C3 denotes the update of the transmission value of each link; C4 denotes that the total capacity of the selected access channels by the transmitter of each link in the current time slot is not greater than the total number of data packets currently needed to be transmitted; constraint condition C5 denotes that only one link can successfully access a channel, otherwise collision occurs; constraint condition C6 gives the mathematical expression of optimization objective Opt2.

[0020] In step 2, based on the Multi-agent deep reinforcement learning framework, N agents are equipped, each agent is equipped with a multi-layer neural network with the same architecture, wherein the first layer is used to receive the local observation of each link, and the last layer directly outputs the Q value of all channel access decisions for action selection.

[0021] The local observation of each agent is represented as:

[0022]

[0023] wherein, denotes the number of currently buffered data packets in all buffer queues of the transmitter of the n-th link; ρn (t) represents the transmission value of the nth link in the current time slot; a n (t-1) represents the decision action of the nth link transmitter in the last time slot; represents the feedback of each channel corresponding to the action of the last time slot; represents the perception result of the nth link to each channel; e and ε are the number of current training and the probability of random selection, respectively.

[0024] The action space of each Agent is defined as:

[0025]

[0026] The dimension is 2 K .

[0027] The framework sets the cooperative reward equation as the benefit of all links and the reward obtained before all link transmitter buffer queues are emptied, and as a constant r0 when all link transmitter buffer queue data packets are emptied:

[0028]

[0029] Step 3 includes:

[0030] Step 3.1: Obtain system parameters, including the number of links N, the number of channels K, the number of buffer queues L, and the maximum number of buffer data packets B of each buffer queue max ;

[0031] Step 3.2: Build a Multi-agent centralized training system based on the Multi-agent deep reinforcement learning framework established in step 2, and perform offline training of all Agent neural network parameters;

[0032] Step 3.3: Deploy the neural network trained in step 3.2 to the transmitters in the actual network;

[0033] Step 3.4: All link transmitters in the network obtain local observations including traffic buffer queue state, latest transmission value, last time slot action, feedback of each channel, and perception result in each time slot, and input them into the loaded Agent neural network model to obtain the channel selection result of the time slot.

[0034] The Multi-agent centralized training system in step 3.2 includes an Agent corresponding to each link, an experience memory replay pool corresponding to each Agent, and a multi-link dynamic channel access simulation environment.

[0035] In step 3.2, each Agent is provided with a target neural network, a training neural network and an experience replay memory pool, and the target neural network and the training neural network have the same neural network structure.

[0036] The method of the present application has the following beneficial effects: in the method, a multi-channel access constraint model with global service priority balancing effect is established based on the service buffer queue condition of each link transmitter, and the benefits of different transmission resources for different link transmission to the whole network are different; a Multi-agent deep learning modeling suitable for multi-channel access considering service priority is performed based on the established model, and the neural network of all Agents is trained using the Multi-agent deep learning model to obtain a distributed method for multi-channel access considering service priority, so that each link can make channel selection decision with global service priority balancing effect only based on local observation when distributed execution. In the method of the present application, each link transmitter can maximize the global benefit only based on local observation when making channel access decision without interaction overhead during actual operation, thereby improving the use efficiency of limited spectrum resources. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A flow chart of a multi-channel access modeling and distributed implementation method considering service priority in the present application;

[0038] Figure 2 A multi-channel access system model considering service priority in the method of the present application;

[0039] Figure 3 A Multi-agent deep reinforcement learning framework based on the method of the present application;

[0040] Figure 4 A comparison chart of the method of the present application and other methods on the global benefit;

[0041] Figure 5 A comparison chart of the method of the present application and other methods on the global total throughput. DETAILED DESCRIPTION

[0042] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings, and it should be noted that the specific embodiments are only the best embodiments of the present application, and do not represent the scope of the present application, and modifications and changes made by those skilled in the art without departing from the main idea of the present application are within the scope of the present application.

[0043] A multi-channel access modeling and distributed implementation method considering service priority is proposed in the present application, as shown in Figure 1The method for modeling a multi-link dynamic channel access method considering service priority is first shown. Based on the multi-agent deep reinforcement learning distributed implementation process, a scheme is provided to solve the problems of low utilization efficiency of wireless resources and lack of global service priority balance awareness in traditional multi-link dynamic spectrum access. The method considers the differences in the service buffer queue states of different link transmitters, proposes a value measurement parameter for measuring the value generated by transmitting different link services on the same wireless resource, and redefines the network transmission benefit based on the parameter. A multi-channel access constraint optimization model considering service priority is established with the defined benefit as the maximization target. By converting the model into a multi-agent deep reinforcement learning process, and using the optimization target of the model as the reward to train all agents, a distributed implementation method of the dynamic multi-channel access considering service priority is obtained, so that the link transmitters loaded with the corresponding agents have the ability of global service priority balance when performing dynamic channel access decisions in a distributed manner. Simulation results show that, under the condition of only using local observation, the results of the method proposed in the application are closer to the optimal solution of the constraint optimization problem, and are significantly better than the random access method.

[0044] Figure 2 The multi-channel access system model considering service priority is established by the application. In the model, N wireless links simultaneously access a wireless spectrum containing K orthogonal channels, wherein the transmitter of each link has a service buffer queue containing L priority levels, and the smaller the buffer queue label, the higher the priority. The maximum buffer capacity of each queue is B max data packets. Each channel is divided into equal period time slots in time, and F consecutive time slots define a time frame. Assuming that all links have completed time synchronization, each time slot on each channel is called a transmission opportunity, and the capacity of all transmission opportunities is 1 data packet. At time slot t, the transmitter of the nth link makes a channel selection decision according to its own buffer queue state to obtain the feedback of each channel wherein indicates that the ith channel is selected by the transmitter of link n, otherwise it is not selected, and indicates that the information of link n is successfully transmitted on the ith channel, otherwise it is not successfully transmitted or there is no feedback because the link is not selected. The transmission value variable of each link is:

[0045]

[0046] wherein, B u is a constant for normalizing the link transmission value, is the weight of the l * priority queue, and has indicates the index of the highest priority queue in the priority queues of the nth link transmitter which is not zero at present.

[0047] The benefit of transmitting different link traffics in each transmission opportunity to the whole network is:

[0048]

[0049] The multi-channel access problem considering traffic priority is modeled as:

[0050]

[0051]

[0052]

[0053] wherein, the optimization target Opt1 represents the benefit sum of all links in a time frame, and the optimization target Opt2 is the time slot index when all links complete transmission in the current time frame. The constraint condition C1 represents the definition of the decision action of each link transmitter; the constraint condition C2 represents the update of data packets in each level traffic buffer queue of each link transmitter; the constraint condition C3 represents the update of transmission value of each link; the constraint condition C4 represents that the total capacity of the access channel selected by each link transmitter in the current time slot is not greater than the total number of data packets currently required to be transmitted; the constraint condition C5 represents that one channel can be successfully accessed by only one link, otherwise collision will occur; and the constraint condition C6 gives the mathematical expression of the optimization target Opt2.

[0054] The optimal solution of the above constraint optimization model can be obtained based on global information, but a large amount of signal overhead is consumed. In order to realize the distributed execution of the above solution, the present application adopts a Multi-agent deep reinforcement learning framework as shown in Figure 3 to obtain a distributed execution method corresponding to the above established model. In the framework, each transmission link corresponds to an Agent, and each Agent is equipped with a multi-layer neural network with the same architecture, wherein the first layer is used to receive the local observation of each link, and the last layer directly outputs the Q value of all channel access decisions for action selection. The deep reinforcement learning algorithm in the present application includes but is not limited to DQN algorithm, Double DQN algorithm and A3C algorithm. The neural network structure of each Agent includes but is not limited to fully connected network, convolutional neural network and recurrent neural network.

[0055] The local observation of each Agent in the framework is represented as:

[0056]

[0057] wherein, represents the number of packets currently buffered in all the buffer queues of the nth link transmitter; p n (t) represents the transmission value of the nth link in the current time slot; a n (t-1) represents the decision action of the nth link transmitter in the previous time slot; represents the feedback of each channel corresponding to the action in the previous time slot; represents the perception result of the nth link to each channel; e and ε are the number of current training and the probability of random selection, respectively. The addition of the three observations e and ε is to cope with the instability problem when training multiple Agents.

[0058] In this framework, the action space of each Agent is defined as:

[0059]

[0060] with a dimension of 2 K .

[0061] The cooperative reward equation set in this framework is as follows:

[0062]

[0063] The physical meaning is that the reward obtained is the sum of the benefits of all links before all the buffer queues of the link transmitters are emptied, and the reward obtained is a constant r0 when all the buffer queues of the link transmitters are emptied. The introduction of r0 is to promote each link to transmit the buffered data packets as soon as possible within the current time frame.

[0064] This framework uses a deep neural network to fit the relationship between the Q value function and the input observation state Q(O n , a n | θ), where θ is the parameter of the neural network, and uses experience replay operation to train the neural network. The neural network parameters are trained offline and loaded onto each link transmitter of the network. In actual use, it is executed in a distributed manner. The centralized training system includes Agents corresponding to each link, experience memory replay pools corresponding to each Agent, and a multi-link dynamic channel access simulation environment. Each Agent is equipped with a target neural network, a training neural network, and an experience replay memory pool. The target neural network and the training neural network have the same neural network structure. The specific steps are as follows:

[0065] Step 1: Obtain system parameters, including the number of links N, the number of channels K, the number of buffer queues L, and the maximum number of buffered packets of each buffer queue;

[0066] Step 2: Based on Figure 3The multi-agent deep reinforcement learning framework shown constructs a centralized training system to perform offline training of all agent neural networks. The specific steps are as follows:

[0067] Step 2.1: Initialize the parameters of 2N neural networks and N experience pools; set the random generation interval for business operations. The number of data packets in each level of the transmitter's buffer queue is randomly initialized according to the interval; the initial observations of each link are set as follows: the number of data packets in each level of the buffer queue in the first time slot, the transmission value of the first time slot, the action selection of the previous time slot are all 0, the channel feedback is all 0, the channel perception result is all 1, the current training count is 1, and the probability of random selection is 1; the greedy probability coefficient ε is set to 1.

[0068] Step 2.2: Begin executing the loop process, which includes N... e The second largest loop contains F smaller loops. N e To ensure the algorithm reaches convergence after a certain number of iterations, every F mini-loops, each agent randomly samples M experiences from its own experience memory replay pool, calculates the mean squared error loss, and trains its neural network parameters using the RMSprop algorithm. Every N mini-loops... p In the second major loop, each agent copies its trained neural network parameters to the target neural network parameters. Where N... p < <N e .

[0069] In this process, the greedy probability coefficient ε is reduced after each major cycle. Also after each major cycle, the random generation interval for the business is reset, and the local observations for each chain are reinitialized.

[0070] The t-th iteration of step 2.2 is as follows:

[0071] Step 2.2.1: Each Agent observes the current environmental state and obtains the local observation state O. n (t);

[0072] Step 2.2.2: Observe O n (t) is input into the training neural network, and a decision action is selected based on the output of the training neural network and the random selection probability:

[0073]

[0074] in, This indicates that an action is selected randomly.

[0075] Step 2.2.3: All agents jointly execute the selected action a n (t), receiving rewards from the environment:

[0076]

[0077] Step 2.2.4: All Agents update the data packet in the buffer queue, the transmission value of itself, the action selection of the last time slot, the channel feedback, the channel observation, the current training times and the probability of random selection, so as to obtain the next observation state O n (t+1).

[0078] Step 2.2.5: Each Agent stores (O n (t),a n (t),r n (t+1),O n (t+1)) into the experience memory replay pool.

[0079] Step 3: deploying the neural network trained in step 2 to the transmitters in the actual network;

[0080] Step 4: the transmitters of all links in the network obtain the local observation including the buffer queue state of the service, the latest transmission value, the action of the last time slot, the feedback of each channel and the sensing result, and input them into the loaded Agent neural network model, so as to obtain the channel selection result of the time slot.

[0081] The application will be described below through specific examples. A dynamic channel access deep learning model containing 6 transmission links and 5 orthogonal channels is trained under the parameter settings in Table 1 by using the method of the application.

[0082] Table 1 shows the example parameter setting table

[0083] Number of channels K 5 Number of links N 6 Number of buffer queue stages L 4 Maximum buffered packets B max ]]> 125 Number of slots F contained in one time frame 100 Capacity of one transmission opportunity 1 data packet Neural network structure Input layer + LSTM layer + fully connected layer + output layer Greediness probability ε Decreasing from 1 to 0.002 by equal difference in the first 80% training rounds Optimizer RMSProp Capacity of experience memory replay pool 20000 Number of samples M 2000 Total number of training rounds 10000

[0084] Figure 4 and Figure 5 The network performance after using the method of the application is compared with the optimal solution and the random selection method. As shown in the figure, the optimal solution can achieve the maximum transmission capacity while optimizing the network benefit defined in the application, which proves the effectiveness of the multi-channel access modeling method considering the service priority proposed in the application. In addition, the distributed implementation method based on Multi-agent deep reinforcement learning proposed in the application has greater overall network benefit improvement and higher overall network throughput compared with the random scheme, and its performance is closer to the optimal solution of the proposed constrained optimization problem under the condition of using only local information, which proves the effectiveness of the distributed implementation method proposed in the application.

[0085] The above embodiments are only the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for modeling and distributed implementation of multi-channel access considering service priority, comprising the following steps: Step 1: establishing a multi-link dynamic channel access constraint optimization model considering service priority, In the model, N wireless links access a wireless spectrum containing K orthogonal channels simultaneously, each transmitter of the links has a buffer queue containing L priorities, and the smaller the queue index, the higher the priority, B max is the maximum number of packets in each buffer queue; each channel is divided into equal periodic slots in time, and F consecutive slots define a time frame; assuming that all links have completed time synchronization, each slot on each channel is called a transmission opportunity, and the capacity of all transmission opportunities is 1 packet; at slot t, the transmitter of the nth link makes a channel selection decision according to its own buffer queue situation Thus, the feedback of each channel is obtained , which indicates that the ith channel is selected by the transmitter of the nth link, otherwise it is not selected, , which indicates that the information of the nth link is successfully transmitted on the ith channel, otherwise it is not successfully transmitted or there is no feedback because the link is not selected; ρ n (t) is a link transmission value variable:​ where B u is a constant for normalizing the link transmission value, is the weight of the l * th priority queue, and has denotes the index of the highest priority queue among the priority queues that are currently non-empty for the n th link transmitter; denotes the number of packets currently buffered in all the buffer queues for the n th link transmitter; The benefit to the network as a whole for each transmission opportunity to transmit different link traffic is: Optimization objective Opt1 represents the benefit sum of all links within a time frame, and optimization objective Opt2 is the time slot number when all links complete transmission within the current time frame; constraint condition C1 represents the definition of transmitter decision action of each link; constraint condition C2 represents the update of data packets in each level of service buffer queue of the transmitter of each link; constraint condition C3 represents the update of transmission value of each link; C4 represents that the total capacity of the access channel selected by the transmitter of each link in the current time slot is not greater than the total number of data packets currently required for transmission; constraint condition C5 represents that a channel can be successfully accessed by only one link, otherwise collision will occur; constraint condition C6 gives the mathematical expression of optimization objective Opt2; Step 2: converting the constraint optimization model established in step 1 into a Multi-agent reinforcement learning decision process, and constructing a deep learning framework for multi-link dynamic channel decision based on the constraint optimization model; Step 3: taking the optimization objective of the constraint optimization model established in step 1 as the reward, and centrally training all Agents of the deep learning framework constructed in step 2, loading the trained Agents into the corresponding transmitters of each link in the network for distributed execution, inputting the local observation of the current time slot, and thereby obtaining the channel selected for access by the transmitter of each link in the current time slot.

2. The method of claim 1, wherein, In step 2, N Agents are provided in the Multi-agent deep reinforcement learning framework, each Agent is provided with a multi-layer neural network of the same architecture, wherein the first layer is used to receive the local observation of each link, and the last layer directly outputs the Q value of all channel access decisions for action selection; the local observation of each Agent is: wherein, ρn(t) represents the number of packets currently buffered in all queues of the transmitter of the nth link; n (t) represents the transmission value of the current time slot of the nth link; a n (t-1) represents the decision action of the transmitter of the nth link in the previous time slot; φn(t) represents the feedback of each channel corresponding to the action in the previous time slot; φn(t) represents the sensing result of each channel of the nth link; e and ε are the number of current training and the probability of random selection, respectively; The action space of each Agent is defined as: Its dimension is 2 K ; The deep reinforcement learning framework sets the cooperative reward equation according to the reward obtained before the buffer queues of all link transmitters are emptied as the benefit sum of all links, and the reward obtained after the data packets in the buffer queues of all link transmitters are emptied as a constant r0:

3. The method of claim 1, wherein, Step 3 includes: Step 3.1: Obtain system parameters, including the number of links N, the number of channels K, the number of buffer queues L, and the maximum number of buffered packets B for each buffer queue max ; Step 3.2: constructing a Multi-agent centralized training system based on the Multi-agent deep reinforcement learning framework established in step 2, and performing offline training of the neural network parameters of all Agents; Step 3.3: deploying the trained neural network in step 3.2 to the transmitters in the actual network; Step 3.4: the transmitters of all links in the network obtain the local observation including the service buffer queue state, the latest transmission value, the action of the last time slot, the feedback of each channel, and the perception result in each time slot, and input the local observation into the loaded Agent neural network model to obtain the channel selection result in the time slot.

4. The method of claim 3, wherein, The multi-agent centralized training system in step 3.2 comprises an agent corresponding to each link, an experience memory replay pool corresponding to each agent, and a multi-link dynamic channel access simulation environment.

5. The method of claim 3, wherein, In step 3.2, each agent is provided with a target neural network, a training neural network and an experience replay memory pool, and the target neural network and the training neural network have the same neural network structure.

6. The method of claim 3, wherein, The offline training in step 3.2 comprises the following steps: Step 3.2.1: initialize the parameters of 2N neural networks and N experience pools; set the service random generation interval According to the interval, the number of data packets in each level of the buffer queue of the transmitter of each link is randomly initialized; the initial observation of each link is that the number of data packets in each level of the buffer queue in the first time slot, the transmission value in the first time slot, the action selection in the last time slot are all 0, the channel feedback is all 0, the channel sensing result is all 1, the current training number is 1 and the randomly selected probability is 1; the greedy probability coefficient ε is set to 1; Step 3.2.2: Start to execute a loop process, the loop includes N e times of large loops, each large loop contains F times of small loops; every F times of small loops, each Agent randomly samples M experiences from the respective experience memory replay pool, calculates the mean square error loss, and trains the respective neural network parameters using the RMSprop algorithm; every N p times of large loops, each Agent copies the respective trained neural network parameters to the target neural network parameters, where N e is such that the algorithm converges within the number of loop times, N p <<N e .

7. The method of claim 6, wherein, After each large cycle, the greedy probability coefficient ε is reduced.

8. The method of claim 6, wherein, After each large cycle, the service random generation interval is reset, and the local observation of each link is reinitialized.

9. The method of claim 6, wherein, The process of the tth small cycle is as follows: (a) each Agent observes the current environment state, obtaining a local observation state O n (t); (b) the observation state O n (t) input into the trained neural network and select a decision action a based on the output of the trained neural network and a randomly selected probability n (t); (c) all Agents jointly perform the selected action a n (t), resulting in a reward for the environment: (d) all the data packets in the Agent's update cache queue, its own transmission value, the last time slot action selection, the channel feedback, the channel observation, the current training number, and the probability of random selection, to obtain the next observation state O n (t+1); (e) each Agent stores (O n (t),a n (t),r n (t+1),O n (t+1)) into an experience memory replay pool.

Images