A multi-base station queuing preamble allocation method based on multi-agent collaboration

Abstract

The present invention discloses a multi-base station queuing preamble allocation method based on multi-agent collaboration. Aiming at the congestion problem that occurs when a large number of agents access randomly, a non-competitive preamble allocation method is proposed in the scenario of multiple base stations and multiple cells. Based on deep reinforcement learning, the devices select preambles in a queued manner, and a federated learning training method is adopted to effectively solve the congestion problem that occurs during competitive access. First, the newly accessed agents are grouped and their priorities are set according to the delay tolerance time; secondly, the agents are reasonably allocated to the idle queue based on the multi-intelligence reinforcement learning algorithm; finally, the federated training method is adopted to synchronously optimize the neural network of each agent through the average optimization of the neural network gradient, and the preamble allocation of each agent is completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a method for allocating leading codes when large-scale machine-type devices of the Internet of Things are randomly accessed. Background Art

[0002] Massive machine type communication (mMTC) is one of the three major application scenarios of the fifth generation of mobile communication technology. Machine type communication is a key technology of the fifth generation new radio (5G NR), and plays a huge role in important and critical application scenarios such as telemedicine, autonomous driving, and intelligent transportation. Machine type communication (MTC), also known as M2M communication, is different from human-to-human (H2H) communication. M2M communication mainly occurs in the uplink, with a large number of terminals, short duration and frequent times. Under the traditional access method, MTC devices always choose the evolved Node B with the best signal quality for access. A large number of MTC devices collide, causing network congestion and seriously affecting the access success rate of devices. Therefore, how to design a reasonable scheme for random access of large-scale MTCD has become the key to 5G mobile communication systems. The most promising solution is to use reinforcement learning to develop a set of preamble allocation schemes, allowing devices to make decisions, select appropriate preambles, and minimize conflicts during random access. In these schemes, devices compete with each other for preambles. When the number of devices increases, conflicts are inevitable, and the access success rate will become lower and lower. Therefore, it is necessary to develop a reasonable preamble allocation scheme to reduce or even avoid conflicts for large-scale MTCD random access. Summary of the invention

[0003] The purpose of the present invention is to provide a multi-base station queuing preamble code allocation method based on multi-agent collaboration. In order to solve the congestion problem caused by random access of massive multi-agents, a non-competitive preamble code allocation method is proposed in the scenario of multiple base stations and multiple cells.

[0004] To achieve the above functions, the present invention designs a multi-base station queuing preamble code allocation method based on multi-agent collaboration. The target area includes a network composed of at least two base stations, each base station includes a preamble code pool, and for each agent accessing the network, the following steps S1 to S3 are executed to complete the preamble code allocation of each agent;

[0005] S1. Group the agents accessing the network according to their service types, calculate the average delay tolerance for each group of agents, and arrange the average delay tolerance of each group of agents in ascending order to obtain a priority set;

[0006] S2. For each group of agents, assign preamble codes to each agent in each group based on the reinforcement learning algorithm;

[0007] Among them, each preamble corresponds to a queue, and the state space S is constructed by the maximum queue number of each queue, and the action space A is constructed by the action of the agent selecting the preamble to queue. With the state space S as input, based on the deep neural network and combined with the Q learning method, the agent selects the action in the action space A as the executable action of the agent based on the greedy strategy and takes the maximization of profit as the goal, and takes the Q value of the executable action of the agent as the output to construct the local agent preamble allocation model;

[0008] S3. Based on the local agent preamble allocation model corresponding to each agent and the federal agent, a global agent preamble allocation model is constructed. Based on the federated learning method, the global agent preamble allocation model is trained to obtain the trained global agent preamble allocation model. The global agent preamble allocation model is applied to complete the preamble allocation of each agent accessing the network.

[0009] As a preferred technical solution of the present invention: in step S1, the agents accessing the network are grouped according to the service type, the average delay tolerance is calculated for each group of agents, and the average delay tolerance of each group of agents is arranged in ascending order. The specific steps of obtaining the priority set are as follows:

[0010] S11: According to the delay requirements of the services of each agent, the similarity of the services of each agent is calculated as follows:

[0011]

[0012] In the formula, c(i,j) is the similarity between business i and business j, t i is the delay requirement of service i, t j is the delay requirement of service j, σ is the similarity coefficient, 0≤c(i,j)≤1;

[0013] According to the similarity of the services of each agent, the services of agents whose similarity difference is less than a preset value are regarded as the same type of services, and the corresponding agents are divided into the same group of agents;

[0014] S12: For each group of agents, the average delay tolerance is calculated as follows:

[0015]

[0016] Where N k represents the number of agents in the kth group, represents the average delay tolerance of the k-th group of agents;

[0017] S13: Calculate the average delay tolerance of each group of agents, expressed as Where n is the number of agent groups. The average delay tolerance of each group of agents is arranged in ascending order and assigned priorities in turn. The priority order is that the agent group with the smallest average delay tolerance is given the highest priority, and the agent group with the largest average delay tolerance is given the lowest priority, and a priority set consisting of the priorities of each agent group is obtained.

[0018] As a preferred technical solution of the present invention: the specific steps of step S2 are as follows:

[0019] S21: Each preamble corresponds to a queue, and the state is constructed based on the maximum queue number of each queue at time t as follows:

[0020] s t ={p 1 ,p 2 ,…,p i ,…,p M}

[0021] In the formula, s t is the state at time t, p i is the maximum queue number of the i-th queue, i∈{1,2,…,M}, M is the total number of queues;

[0022] The state space S is constructed from the state from the initial time to time t as follows:

[0023] S={s0,s1,…,s t}

[0024] Among them, s0, s1, …, s t Represents the state from the initial moment to the moment t, s0 is the state at the initial moment;

[0025] S22: When the agent accesses the network, it selects one from the queues corresponding to the M preamble codes to queue up, and the action space A is constructed by the action of the agent selecting the preamble code to queue up as follows:

[0026] A={a 1 ,a 2 ,…,a i ,…,a M}

[0027] In the formula, a i represents the action strategy of the agent, i.e., the action of selecting the i-th preamble for queuing;

[0028] S23: Action strategy a selected by the agent 1 ,a 2 ,…,a n , corresponding to the reward r 1 ,r 2 ,…,rn , and construct the reward function R as follows:

[0029] R=r i (r 1 ,r 2 ,…,r n )

[0030] Introducing the priority of the agent and the variance of each queue, the reward function R is transformed into the following form:

[0031]

[0032] In the formula, f i (a 1 ,a 2 ,…,a n ) represents the priority of agent i, g i (a 1 ,a 2 ,…,a n ) represents the variance of the cohort;

[0033] S24: Based on deep neural network and combined with Q learning method, a local agent preamble allocation model is constructed, with state space S as input and Q value of the agent's executable action as output. t Each action in the state corresponds to a Q value Q(s t ,a t ), where a t The specific formula is as follows:

[0034]

[0035] In the formula, a represents the state s t All executable actions are listed below;

[0036] According to the Q learning algorithm, the Q value Q at the next moment is updated by the following formula k+1 (s t ,a t ):

[0037]

[0038] In the formula, α k and γ are the learning rate and discount factor respectively, s t+1 represents the state at the next moment, r t+1 Indicates that in state s t+1 The reward obtained by the agent's executable action under the state s t+1 The executable actions of the agent, A is the action space, Q k (s t ,a t) indicates state s t Q value under max a′∈A Q k (s t+1 ,a′) represents the state s t+1 The maximum Q value corresponding to each executable action in the action space A;

[0039] S25: Update status s t+1 and its corresponding reward r t+1 , construct the experience sample (s t ,a t ,s t+1 ,r t+1 ) and store it in the experience database;

[0040] S26: Loss function L of each agent i (θ) is as follows:

[0041] L i (θ)=E[(y DQN -Q k (s t ,a t ;θ)) 2 ]

[0042] In the formula, θ represents the weight of the online network;

[0043] Among them, y DQN The calculation is as follows:

[0044]

[0045] In the formula, a′ i represents the action that maximizes the Q value of the target network in state s′, θ - represents weight;

[0046] S27: Randomly extract experience samples from the experience library to train the local agent preamble code allocation model.

[0047] As a preferred technical solution of the present invention: in step S2, the local agent preamble code allocation model is trained for a preset number of times before the state is updated.

[0048] As a preferred technical solution of the present invention: in step S2, each agent uses the ε greedy strategy to select action a i , select the action strategy in the action space A with the probability of exploration factor ε, and select the best action strategy in the action space A with the probability of (1-ε).

[0049] As a preferred technical solution of the present invention: in step S3, the specific steps of training the global agent preamble allocation model based on the federated learning method are as follows:

[0050] S31: All agents select action strategies based on their current states and receive corresponding rewards;

[0051] S32: Each agent inputs the current state into the deep neural network in the preamble code allocation model of its own local agent for learning, obtains the parameters of the preamble code allocation model of each local agent, and sends them to the federated agent;

[0052] S33: The federated agent adopts an aggregate average algorithm to learn the parameters of the preamble code allocation model of each local agent to obtain the global agent preamble code allocation model, where the parameters of the global agent preamble code allocation model are as follows:

[0053]

[0054] In the formula, θ g Assign model weights, θ, to the global agent preamble l Assign model weights to the local agent preamble, D is the number of training data, and D k Indicates the amount of data owned by the kth participant.

[0055] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0056] (1) Different from the traditional competitive preamble method, non-competitive queuing access can solve the collision problem and allow more intelligent agents to access under the same conditions.

[0057] (2) When the intelligent agent in the present invention makes a decision, it adopts a multi-agent reinforcement learning algorithm to collaboratively select a suitable preamble code. The learning algorithm can better adapt to environmental changes and make the best decision.

[0058] (3) Using federated learning for training can improve the performance of reinforcement learning and train a more robust model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a schematic diagram of agent grouping provided according to an embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of an intelligent agent accessing a network according to an embodiment of the present invention;

[0061] Figure 3 is a diagram of an intelligent agent neural network structure provided according to an embodiment of the present invention;

[0062] Figure 4 It is a diagram of a federated training model provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0064] An embodiment of the present invention provides a multi-base station queuing preamble code allocation method based on multi-agent collaboration, wherein a target area includes a network composed of at least two base stations, each base station includes a preamble code pool, and for each agent accessing the network, the following steps S1 to S3 are executed to complete the preamble code allocation of each agent;

[0065] S1. The agents are MTC devices, and each agent has its own service type. According to the service type of each agent, the agents accessing the network are grouped. The average delay tolerance is calculated for each group of agents, and the average delay tolerance of each group of agents is arranged in ascending order to obtain a priority set. The agent grouping diagram is shown in Figure 1 ;

[0066] The specific steps of step S1 are as follows:

[0067] S11: There are different types of services in the network, which are divided into delay-tolerant services and delay-sensitive services according to the different sensitivity of each type of service to delay. In addition, the QoS requirements of each intelligent agent need to be considered. Due to the large number of intelligent agents accessing the network at the same time, the types of services accessed at the same time are also different. In view of the current network requirements for MTC applications, the delay requirements of each type of service are used to measure the correlation of the service types. According to the delay requirements of the services of each intelligent agent, the similarity of the services of each intelligent agent is calculated as follows:

[0068]

[0069] In the formula, c(i,j) is the similarity between business i and business j, t i is the delay requirement of service i, t j is the delay requirement of service j, σ is the similarity coefficient, 0≤c(i,j)≤1, the larger the c(i,j), the more similar the two services are;

[0070] According to the similarity of the services of each agent, the services of agents whose similarity difference is less than a preset value are regarded as the same type of services, and the corresponding agents are divided into the same group of agents;

[0071] S12: For each group of agents, the average delay tolerance is calculated as follows:

[0072]

[0073] Where N k represents the number of agents in the kth group, represents the average delay tolerance of the k-th group of agents;

[0074] S13: Calculate the average delay tolerance of each group of agents, expressed as Where n is the number of agent groups. The average delay tolerance of each group of agents is arranged in ascending order and assigned priorities in turn. The priority order is that the agent group with the smallest average delay tolerance is given the highest priority, and the agent group with the largest average delay tolerance is given the lowest priority, and a priority set consisting of the priorities of each agent group is obtained.

[0075] S2. For each group of agents, the preamble code is assigned to each agent in each group based on the reinforcement learning algorithm. Using the idea of ​​reinforcement learning, the agent continuously interacts with the network and selects the action that can maximize the benefit based on the network;

[0076] Reinforcement learning is used to solve the problem of Markov decision process. In reinforcement learning, the agent can learn to take actions periodically, observe the maximum benefit and automatically adjust the action strategy to obtain the optimal action strategy. Since the agents are grouped, multiple agents learn in the interaction with the network. In the case of competitive games, multi-agents can reach local optimality, but cannot meet the overall network performance maximization. In order to achieve the goal of optimization problem, the multi-agent problem is transformed into a cooperative game, using the same reward function for all agents.

[0077] Among them, each preamble corresponds to a queue, and the state space S is constructed by the maximum queue number of each queue. The action space A is constructed by the action of the agent selecting the preamble to queue. With the state space S as input, based on the deep neural network and combined with the Q learning method, the agent selects the action in the action space A as the executable action of the agent based on the greedy strategy and takes the maximization of profit as the goal. The Q value of the executable action of the agent is used as the output to construct the local agent preamble allocation model. The schematic diagram of the agent accessing the network is referred to Figure 2 , in the figure, R1, R2, …, R M-1 ,R M Indicates the leading code;

[0078] The specific steps of step S2 are as follows:

[0079] S21: Each preamble corresponds to a queue, and the state is constructed based on the maximum queue number of each queue at time t as follows:

[0080] s t ={p 1 ,p 2 ,…,p i ,…,p M}

[0081] In the formula, s t is the state at time t, p i is the maximum queue number of the i-th queue, i∈{1,2,…,M}, M is the total number of queues;

[0082] The state space S is constructed from the state from the initial time to time t as follows:

[0083] S={s0,s1,…,s t}

[0084] Among them, s0, s1, …, s t Represents the state from the initial moment to the moment t, s0 is the state at the initial moment;

[0085] S22: When the agent accesses the network, it selects one from the queues corresponding to the M preamble codes to queue up, and the action space A is constructed by the action of the agent selecting the preamble code to queue up as follows:

[0086] A={a 1 ,a 2 ,…,a i ,…,a M}

[0087] In the formula, a i represents the action strategy of the agent, i.e., the action of selecting the i-th preamble for queuing;

[0088] S23: Action strategy a selected by the agent 1 ,a 2 ,…,a n , corresponding to the reward r 1 ,r 2 ,…,r n , and construct the reward function R as follows:

[0089] R=r i (r 1 ,r 2 ,…,r n )

[0090] When the number of agents in each queue tends to be consistent, there is no idle queue, so there is no idle preamble, and the access efficiency is higher. When there are a large number of agents, agents with high priority will enter the queue faster and access within the delay tolerance time, ensuring the access success rate of the agents.

[0091] Introducing the priority of the agent and the variance of each queue, the reward function R is transformed into the following form:

[0092]

[0093] In the formula, f i (a 1 ,a 2 ,…,a n ) represents the priority of agent i. The agent with the highest priority enters the queue and receives the largest reward. i (a 1 ,a 2 ,…,a n ) represents the variance of the cohort;

[0094] S24: Based on the deep neural network and combined with the Q learning method, a local agent preamble allocation model is constructed, with the state space S as input and the Q value of the agent's executable action as output. The neural network structure diagram of the agent is shown in Figure 3 , the agent is in s t Each action in the state corresponds to a Q value Q(s t ,a t ), where a t The specific formula is as follows:

[0095]

[0096] In the formula, a represents the state s t All executable actions are listed below;

[0097] According to the Q learning algorithm, the Q value Q at the next moment is updated by the following formula k+1 (s t ,a t ):

[0098]

[0099] In the formula, α k and γ are the learning rate and discount factor respectively, s t+1 represents the state at the next moment, r t+1 Indicates that in state s t+1 The reward obtained by the agent's executable action under the state s t+1 The executable actions of the agent, A is the action space, Q k (s t ,a t ) indicates state s t Q value under max a′∈A Q k (s t+1 ,a′) represents the state s t+1 The maximum Q value corresponding to each executable action in the action space A;

[0100] S25: Update status s t+1 and its corresponding reward rt+1 , construct the experience sample (s t ,a t ,s t+1 ,r t+1 ) and store it in the experience database;

[0101] S26: Loss function L of each agent i (θ) is as follows:

[0102] L i (θ)=E[(y DQN -Q k (s t ,a t ;θ)) 2 ]

[0103] In the formula, θ represents the weight of the online network;

[0104] Among them, y DQN The calculation is as follows:

[0105]

[0106] In the formula, a′ i represents the action that maximizes the Q value of the target network in state s′, θ - represents weight;

[0107] S27: Randomly extract experience samples from the experience library to train the local agent preamble code allocation model.

[0108] In one embodiment, in step S2, the local agent preamble code allocation model is trained for a preset number of times before the status is updated.

[0109] In one embodiment, in step S2, each agent uses the ε greedy strategy to select action a i , select the action strategy in the action space A with the probability of exploration factor ε, and select the best action strategy in the action space A with the probability of (1-ε).

[0110] S3. Based on the local agent preamble allocation model corresponding to each agent and the federated agent, a global agent preamble allocation model is constructed. Based on the federated learning method, the global agent preamble allocation model is trained to obtain the trained global agent preamble allocation model. The global agent preamble allocation model is applied to complete the preamble allocation of each agent accessing the network. The federated training model diagram is referenced Figure 4 .

[0111] Since individual agents in a multi-agent system face different tasks or situations, the experience samples stored in the experience library cannot adapt to changes. Therefore, a federated training method is adopted to synchronously optimize the neural network of each agent through the average optimization of the neural network gradient. In this federated training method, each agent optimizes its own neural network through local experience and the neural network gradients from other collaborative agents. A federated agent is designed with the goal of collecting various local gradients of the agents involved and optimizing them on average. This federated agent has the same neural network structure as other agents, but does not take any actions.

[0112] The specific steps of training the global agent preamble allocation model based on the federated learning method in step S3 are as follows:

[0113] S31: All agents select action strategies based on their current states and receive corresponding rewards;

[0114] S32: Each agent inputs the current state into the deep neural network in the preamble code allocation model of its own local agent for learning, obtains the parameters of the preamble code allocation model of each local agent, and sends them to the federated agent;

[0115] S33: The federated agent adopts an aggregate average algorithm to learn the parameters of the preamble code allocation model of each local agent to obtain the global agent preamble code allocation model, where the parameters of the global agent preamble code allocation model are as follows:

[0116]

[0117] In the formula, θ g Assign model weights, θ, to the global agent preamble l Assign model weights to the local agent preamble, D is the number of training data, and D k Indicates the amount of data owned by the kth participant.

[0118] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multi-base station queuing preamble code allocation method based on multi-agent collaboration, characterized in that: The target area includes a network composed of at least two base stations, each base station includes a preamble pool, and for each intelligent agent accessing the network, the following steps S1 to S3 are executed to complete the preamble allocation of each intelligent agent; S1. Group the agents accessing the network according to their service types, calculate the average delay tolerance for each group of agents, and arrange the average delay tolerance of each group of agents in ascending order to obtain a priority set; The specific steps of step S1 are as follows: S11: According to the delay requirements of the services of each agent, the similarity of the services of each agent is calculated as follows: In the formula, c(i,j) is the similarity between business i and business j, t i is the delay requirement of service i, t j is the delay requirement of service j, σ is the similarity coefficient, 0≤c(i,j)≤1; According to the similarity of the services of each agent, the services of agents whose similarity difference is less than a preset value are regarded as the same type of services, and the corresponding agents are divided into the same group of agents; S12: For each group of agents, the average delay tolerance is calculated as follows: Where N k represents the number of agents in the kth group, represents the average delay tolerance of the k-th group of agents; S13: Calculate the average delay tolerance of each group of agents, expressed as Where n is the number of agent groups. The average delay tolerance of each group of agents is arranged in ascending order and assigned priorities in turn. The priority order is that the agent group with the smallest average delay tolerance is given the highest priority, and the agent group with the largest average delay tolerance is given the lowest priority, and a priority set consisting of the priorities of each agent group is obtained; S2. For each group of agents, assign preamble codes to each agent in each group based on the reinforcement learning algorithm; Among them, each preamble corresponds to a queue, and the state space S is constructed by the maximum queue number of each queue, and the action space A is constructed by the action of the agent selecting the preamble to queue. With the state space S as input, based on the deep neural network and combined with the Q learning method, the agent selects the action in the action space A as the executable action of the agent based on the greedy strategy and takes the maximization of profit as the goal, and takes the Q value of the executable action of the agent as the output to construct the local agent preamble allocation model; The specific steps of step S2 are as follows: S21: Each preamble corresponds to a queue, and the state is constructed based on the maximum queue number of each queue at time t as follows: s t ={p 1 ,p 2 ,…,p i ,…,p M } In the formula, s t is the state at time t, p i is the maximum queue number of the i-th queue, i∈{1,2,…,M}, M is the total number of queues; The state space S is constructed from the state from the initial time to time t as follows: S={s0,s1,…,s t } Among them, s0, s1, …, s t Represents the state from the initial moment to the moment t, s0 is the state at the initial moment; S22: When the agent accesses the network, it selects one from the queues corresponding to the M preamble codes to queue up, and the action space A is constructed by the action of the agent selecting the preamble code to queue up as follows: A={a 1 ,a 2 ,…,a i ,…,a M } In the formula, a i represents the action strategy of the agent, i.e., the action of selecting the i-th preamble for queuing; S23: Action strategy a selected by the agent 1 ,a 2 ,…,a n , corresponding to the reward r 1 ,r 2 ,…,r n , and construct the reward function R as follows: R＝r i (r 1 ,r 2 ,…,r n ) Introducing the priority of the agent and the variance of each queue, the reward function R is transformed into the following form: In the formula, f i (a 1 ,a 2 ,…,a n ) represents the priority of agent i, g i (a 1 ,a 2 ,…,a n ) represents the variance of the cohort; S24: Based on deep neural network and combined with Q learning method, a local agent preamble allocation model is constructed, with state space S as input and Q value of the agent's executable action as output. t Each action in the state corresponds to a Q value Q(s t ,a t ), where a t The specific formula is as follows: In the formula, a represents the state s t All executable actions are listed below; According to the Q learning algorithm, the Q value Q at the next moment is updated by the following formula k+1 (s t ,a t ): In the formula, α k and γ are the learning rate and discount factor respectively, s t+1 represents the state at the next moment, r t+1 Indicates that in state s t+1 The reward obtained by the executable action of the agent under ′ Indicates state s t+1 The executable actions of the agent, A is the action space, Q k (s t ,a t ) indicates state s t Q value under max a′∈A Q k (s t+1 ,a ′ ) indicates state s t+1 The maximum Q value corresponding to each executable action in the action space A; S25: Update status s t+1 and its corresponding reward r t+1 , construct the experience sample (s t ,a t ,s t+1 ,r t+1 ) and store it in the experience database; S26: Loss function L of each agent i (θ) is as follows: L i (θ)=E[(y DQN -Q k (s t ,a t ;i)) 2 ] In the formula, θ represents the weight of the online network; Among them, y DQN The calculation is as follows: In the formula, a i ′ Indicates that in state s ′ The action that maximizes the Q value of the target network, θ - represents weight; S27: randomly extracting experience samples from the experience library to train the local agent preamble allocation model; S3. Based on the local agent preamble allocation model corresponding to each agent and the federal agent, a global agent preamble allocation model is constructed. Based on the federated learning method, the global agent preamble allocation model is trained to obtain the trained global agent preamble allocation model. The global agent preamble allocation model is applied to complete the preamble allocation of each agent accessing the network.

2. A multi-base station queuing preamble code allocation method based on multi-agent collaboration as claimed in claim 1, characterized in that: In step S2, the local agent preamble allocation model is trained for a preset number of times before updating the state.

3. A multi-base station queuing preamble code allocation method based on multi-agent collaboration as claimed in claim 1, characterized in that: In step S2, each agent uses the ε greedy strategy to select action a i , select the action strategy in the action space A with the probability of exploration factor ε, and select the best action strategy in the action space A with the probability of (1-ε).

4. A multi-base station queuing preamble code allocation method based on multi-agent collaboration as claimed in claim 1, characterized in that: The specific steps of training the global agent preamble allocation model based on the federated learning method in step S3 are as follows: S31: All agents select action strategies based on their current states and receive corresponding rewards; S32: Each agent inputs the current state into the deep neural network in the preamble code allocation model of its own local agent for learning, obtains the parameters of the preamble code allocation model of each local agent, and sends them to the federated agent; S33: The federated agent adopts an aggregate average algorithm to learn the parameters of the preamble code allocation model of each local agent to obtain the global agent preamble code allocation model, where the parameters of the global agent preamble code allocation model are as follows: In the formula, θ g Assign model weights, θ, to the global agent preamble l Assign model weights to the local agent preamble, D is the number of training data, and D k Indicates the amount of data owned by the kth participant.

Images