AP synchronization method based on deep reinforcement learning for cellular-free large-scale MIMO system

Abstract

The invention relates to a deep reinforcement learning-based AP synchronization method for a cellular-free large-scale MIMO system, and belongs to the field of wireless communication systems. By constructing a system model, a state space, an action space and a reward function and adopting a depth deterministic policy gradient algorithm DDPG to train an agent model, real-time compensation and optimization of AP time-frequency deviation are realized. According to the method, the best action is selected according to the current state through the proxy model, the time-frequency deviation compensation factor of the AP is continuously learned and adjusted by taking the total spectral efficiency gain as an award, and the total spectral efficiency is calculated by using the SINR sensing technology, so that the synchronization precision and the spectral efficiency of the cellular-free large-scale MIMO system are remarkably improved, the network performance is optimized, and the network performance is improved. The method is suitable for the requirements of high-density and high-capacity wireless communication networks in the future.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication systems and relates to an AP synchronization method for a non-cellular large-scale MIMO system based on deep reinforcement learning. Background Art

[0002] In a non-cellular massive MIMO system, ensuring synchronization between access points (APs) is one of the key technologies to maintain the normal operation of the system. The clock oscillator in the AP is responsible for generating the clock signal that controls the data transmission rate. However, due to imperfections in the manufacturing process, temperature changes, power supply voltage fluctuations, aging, and environmental factors, these clock oscillators may not be able to generate accurate clock signals, causing time and frequency deviations between APs in wireless communications, which has a significant impact on the communication efficiency of the system. Synchronization errors not only reduce the quality of data transmission, but may also cause network paralysis. However, traditional synchronization solutions often find it difficult to accurately estimate the time and frequency synchronization deviations between APs caused by hardware.

[0003] In order to solve the problem of time-frequency synchronization between APs, this paper proposes an innovative synchronization algorithm solution based on deep reinforcement learning and combined with signal to interference plus noise ratio (SINR) perception technology. The intelligent decision-making ability of deep reinforcement learning is used to effectively estimate the time-frequency deviation between APs and compensate for the time-frequency deviation, significantly improving the total spectrum efficiency of the non-cellular large-scale MIMO system. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an AP synchronization method based on deep reinforcement learning for a cellular-free massive MIMO system.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The AP synchronization method based on deep reinforcement learning for a cell-free massive MIMO system includes the following steps:

[0007] S1: Establish a system model, including the number of APs, the number of users, the channel model, and the signal model;

[0008] S2: construct the state space, including AP time-frequency deviation compensation factor and anchor AP estimation channel;

[0009] S3: Constructing the action space, including the AP time-frequency deviation compensation factor;

[0010] S4: define a reward function, taking the total spectrum efficiency gain as the reward;

[0011] S5: The deep deterministic policy gradient algorithm DDPG is used to train the agent. The agent selects actions based on the current state and receives rewards from the environment to continuously learn the optimal time-frequency deviation compensation strategy.

[0012] S6: Use the trained proxy model to adjust the AP time-frequency deviation compensation factor in real time to achieve synchronization between APs and improve the overall spectrum efficiency of the system.

[0013] Furthermore, in S1, the system model includes L single-antenna APs and K single-antenna users, the APs and users are connected via channels, and the channel model is a fast fading channel model.

[0014] Furthermore, in S2, the state space includes the cumulative compensation factors of the time-frequency deviations of the N groups of APs and the estimated channel sets of the N anchor APs.

[0015] Furthermore, in S3, the action space is an N-dimensional continuous space, representing the time-frequency deviation compensation factors of N groups of APs.

[0016] Furthermore, in S4, the reward function is the gain of the total spectral efficiency after executing the action.

[0017] Further, in S5, the DDPG algorithm includes an Actor network, an Actor target network, a Critic network, and a Critic target network, and stores the experience data generated by the interaction between the agent and the environment through an experience replay pool.

[0018] Furthermore, in S6, the proxy model adjusts the AP time-frequency deviation compensation factor in real time, and calculates the total spectrum efficiency using SINR perception technology to optimize system performance.

[0019] Further, in S6, the AP time-frequency deviation compensation factor satisfies:

[0020] represents the time-frequency deviation compensation factor of the nth anchor point AP at the tth time; represents the conjugate of the time-frequency deviation of the nth anchor point AP at the current moment;

[0021] At time t, the N-dimensional action for compensating the time-frequency deviation of each group of AP estimated channels is defined as:

[0022]

[0023] in, Represents the exponential value of the time-frequency deviation compensation factor of the nth anchor point AP at the tth time.

[0024] Further, in S6, the equivalent channel after the AP channel time-frequency deviation compensation satisfies:

[0025] When the agent gets the action After that, the channels of all APs Perform time-frequency deviation compensation; the equivalent channel after time-frequency deviation compensation on the i-th AP is expressed as in, represents the equivalent channel between the i-th AP and the k-th user, then action a is executed t After that, for the i-th AP, the channel in the next state is Right now

[0026]

[0027] Among them, s t (n) and a t (n) are the state and action of the nth AP at the tth moment.

[0028] Further, in S6, the total spectrum efficiency based on the SINR perception algorithm is Defined in state s t The measured SINR of the kth user is expressed as The total spectrum efficiency based on the SINR perception algorithm is:

[0029]

[0030] Among them, τ p is the pilot length; τ c is the total signal length; K is the number of users; To obtain the expected operation.

[0031] The beneficial effects of the present invention are:

[0032] (1) By adopting deep reinforcement learning technology, the present invention can compensate for the time and frequency deviation of AP in the cellular-free large-scale MIMO system in real time and accurately, thereby improving the synchronization accuracy of the system.

[0033] (2) The present invention significantly improves the overall spectrum efficiency of the system by optimizing the AP's time-frequency deviation compensation factor, thereby effectively increasing the data transmission rate and system capacity of the network.

[0034] (3) By utilizing the deep deterministic policy gradient algorithm (DDPG), the present invention can stably learn and make optimal decisions in a constantly changing environment, thereby improving the stability and reliability of the system.

[0035] (4) The method of the present invention can adapt to APs and users of different orders of magnitude, as well as different channel and signal models, and has wide applicability.

[0036] (5) The algorithm of the present invention can complete the training of the proxy model in a relatively short time, and can respond quickly and adjust the compensation strategy in real time in practical applications.

[0037] (6) The synchronization method of the present invention does not require complicated hardware changes and can be implemented in the existing non-cellular massive MIMO system through software upgrade, thereby reducing the implementation cost.

[0038] (7) The method of the present invention is adapted to the development needs of future high-density and high-capacity wireless communication networks, and provides technical support for new generation communication technologies such as 5G and 6G.

[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 The overall process of time-frequency deviation compensation;

[0042] Figure 2 This is the flow chart of the DDPG algorithm. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0045] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0046] The present invention fully considers the needs of real-time communication, regards the AP's time-frequency synchronization decision process as a Markov decision process, and optimizes the overall spectrum efficiency based on deep reinforcement learning theory. In traditional reinforcement learning algorithms, the Q table is used to store the reward value of each state-action pair. The Q learning algorithm can theoretically converge to the optimal strategy by exploring the environment and learning from it. Deep reinforcement learning combines the advantages of deep learning and reinforcement learning, and uses deep neural networks to approximate the Q function, namely the Deep Q-Network (DQN), which enables the algorithm to handle Q learning problems in high-dimensional state-action space without clear state transition probabilities.

[0047] In order to further improve the performance of the DQN algorithm, the present invention introduces the Deep Deterministic Policy Gradient (DDPG) algorithm. DDPG introduces the Actor-Critic framework based on DQN, in which the Actor network is responsible for outputting deterministic actions, while the Critic network evaluates the value of these actions. Through this mechanism, DDPG uses deep neural networks to approximate the policy network and value network, and uses the policy gradient method to train network parameters.

[0048] The present invention adopts the DDPG algorithm combined with SINR perception technology to estimate and compensate for the time-frequency deviation of multiple APs. The CPU is used as an agent to improve the overall spectrum efficiency of the system. Through continuous interaction with the environment, the optimal time-frequency deviation compensation strategy is trained and learned in real time. This method can not only improve the synchronization accuracy of the system, but also enhance the adaptability and robustness of the system, thereby achieving more efficient and stable wireless communication in real-time communication scenarios.

[0049] 1. System Model

[0050] In a non-cellular massive MIMO system, a large number of APs work together to serve users, and each AP is connected to the central processing unit (CPU) through a fronthaul link. Assume that the system is configured with L single-antenna APs and K single-antenna users (L>K), and the channel between the l=1,2,…,Lth AP and the k=1,2,…,Kth user is denoted as h kl ,and R kl is the spatial correlation coefficient, assuming that g kl is small-scale fading, β kl is the large-scale fading coefficient, then the standard fast fading channel model is:

[0051]

[0052] Among them, PL lk is the path loss coefficient, is the shadow fading coefficient, σ sh is the standard deviation,

[0053] In a cell-free massive MIMO system, the uplink data consists of pilot and payload data. Assuming that τ p Pilot The length of each pilot is τ p , that is, ||φ t || 2 =τ p , the length of each resource block is τ c .

[0054] The received pilot signal of the lth AP is

[0055]

[0056] Among them, p k ≥0 is the transmission power of the kth user, is Gaussian noise, obey σ 2 is the noise power.

[0057] The received signal Z l The corresponding normalized pilot signal Do relevant and get The noise term is defined as Then Z in formula (3) l Convert to z kl :

[0058]

[0059] definition Using the pilot signal, we can get h under the MMSE criterion. kl The estimated channel for:

[0060]

[0061] 2. Signal Model

[0062] In a non-cellular massive MIMO system, the received signal at the CPU end based on a fully centralized architecture is:

[0063]

[0064] CPU receives the signal vector y=[y1,...,y L ] T It is expressed as:

[0065]

[0066] The channel vector is h k =[h k1 ,...,h kL ] T , is the correlation matrix, R k1 is the correlation coefficient of the channel between the kth user and the lth AP; n = [n k1 ,...,n kl ] T is the noise vector.

[0067] The estimated channel of the kth user is defined as Using MR combiner but In downlink transmission, the SINR of the kth user is:

[0068]

[0069] The spectral efficiency of the kth user is:

[0070]

[0071] The overall spectral efficiency is:

[0072]

[0073] The above process does not consider the time-frequency deviation between APs. When the influence of time-frequency deviation is considered, the exponential form of the time-frequency deviation of the lth AP is expressed as:

[0074]

[0075] Among them, f s is the sampling frequency, B is the bandwidth, f l is the frequency deviation of the lth AP, τ l is the timing deviation of the lth AP. After considering the time-frequency deviation between APs, the received signal at the CPU end in formula (6) becomes:

[0076]

[0077] Received signal vector It is expressed as:

[0078]

[0079] Among them, the channel with time-frequency deviation of the kth user is for:

[0080]

[0081] in, is the time-frequency deviation vector of all APs, is the time-frequency deviation of the lth AP.

[0082] definition for The estimated value of the channel The estimated channel for:

[0083]

[0084] Then the SINR of the kth user is:

[0085]

[0086] In time-frequency deviation Under the influence of

[0087] 3. AP synchronization algorithm based on deep reinforcement learning

[0088] 3.1 Problem Modeling

[0089] DDPG learns the optimal strategy through the interaction between the agent and the environment, where the agent selects actions based on the current state, and the environment influences the agent's behavior through reward feedback. This process involves four key elements: (S, A, R, S′), namely state, action, reward, and next state. The state describes the current situation of the environment, the action is the decision made by the agent to influence the environment or achieve a goal, the reward is the immediate feedback of the environment to the action, and the next state is the new state of the environment after the action is executed. Through continuous trial and error, the agent learns to choose the best action in different states to maximize the cumulative reward.

[0090] In addition, the concept of experience replay pool is introduced in the DDPG algorithm. Used to store experience data generated during the interaction between the agent and the environment. This data is used to break the correlation between samples and allow the algorithm to learn from past experience. By randomly sampling data from the experience replay pool for training, the efficiency and stability of learning can be improved. The capacity of the experience replay pool is limited. When the capacity limit is reached, the new experience will replace the oldest experience to keep the data in the pool fresh. This mechanism helps the algorithm generalize better and reduces the overfitting problem caused by reusing the same experience.

[0091] DDPG consists of four main networks: Critic network, Critic target network, Actor network and Actor target network. The parameters of these four networks are w, w′, θ, θ′ respectively. The functions of these four networks in DDPG are:

[0092] (1) Actor network: takes the current state of the environment as input and outputs a deterministic action, which will be evaluated by the Critic network for its Q value;

[0093] (2) Critic network: used to evaluate the actions output by the Actor network. It takes a state-action pair as input and outputs a Q value, which is used to evaluate the expected return of the current state-action pair. The larger the Q value, the greater the total return that the action is expected to bring.

[0094] (3) Critic target network and Actor target network: Critic network and Actor network are used as main networks. The target network structure is the same as the corresponding main network, but its parameters are updated more slowly. The target network uses a soft update mechanism to slowly track the parameters of the main network to achieve target network parameter updates:

[0095]

[0096] Among them, ξ<<1 is the soft update coefficient of the target network, which is used to control the update speed of the target network. The larger the value, the faster the target network updates. The parameters of the target network will be soft updated regularly or gradually to gradually approach the parameters of the main network to improve the stability of learning.

[0097] The critic training network outputs the Q value of the current state-action pair: Used to evaluate the current strategy, the Q value is calculated as follows:

[0098]

[0099] The critic target network outputs the Q value of the state-action pair at the next moment: Among them, the next action value π θ′ (s t+1 ) is estimated by the Actor target network. The target value of the Q value function in the current state is:

[0100]

[0101] The Critic training network updates the parameters of the network by minimizing the loss function, which is defined as:

[0102]

[0103] Where m is the number of samples taken from the experience replay pool each time, (s i ,a i )from A batch of m state-action pairs taken from .

[0104] Actor training network provides the strategy π for the current state θ (s t ), the Actor target network provides the next state strategy π θ′ (s t+1 ). Combining the Q value function of the Critic training network, the policy gradient of the Actor training network can be obtained:

[0105]

[0106] The parameter update of the Critic target network and the Actor target network refers to formula (18).

[0107] To increase the randomness of the agent's exploration, actions during training will be added with an exploration mechanism:

[0108]

[0109] Where ε∈(0,1) is the exploration rate. Formula (23) means that a random action is executed with probability ε, and a random action is executed with probability (1-ε) according to the strategy π θ (s i ) to perform an action.

[0110] 3.2 AP Time-Frequency Synchronization Algorithm Based on DDPG

[0111] Since there are a large number of APs in the non-cellular massive MIMO system, if all APs are optimized, the training complexity will be very high and difficult to converge. In order to reduce the complexity of the system, the present invention adopts the "local master-slave" concept, that is, among all APs, a clustering algorithm is used to divide the APs into N groups according to their location distance. In each group, the AP closest to the cluster center is selected as the anchor AP, and the others are ordinary APs. The set of all APs is defined as u n is the nth group of AP sets; each AP subset is Among them, g n =0 indicates anchor point AP, g n =1,2,...,G n Indicates a common AP, the number of APs in the group is G n Determined by the clustering algorithm. Define the anchor AP subset as

[0112] The present invention introduces SINR perception technology. The CPU agent uses SINR measurement feedback information to calculate the total spectrum efficiency and adjust the time-frequency deviation in real time to optimize the total spectrum efficiency. The overall idea of ​​the algorithm is as follows: Figure 1 .

[0113] Based on the DDPG framework, the states, actions, and rewards of the AP time-frequency synchronization algorithm are defined as follows:

[0114] (1) Status

[0115] Assuming that the value of the time-frequency deviation compensation factor of all APs is 1 at the initial time t=1, the state of the nth group of APs at the tth time is:

[0116]

[0117] Define the N-dimensional state of the cumulative compensation of the time-frequency deviation of N groups of APs at time t as:

[0118] s t =[s t (1),...,s t (N)] T (25)

[0119] The estimated channel set of the nth anchor AP at time t is defined as in, is the estimated channel between the kth user and the nth anchor AP in the time-frequency deviation scenario Its definition refers to formula (5). is the estimated channel at the initial time t = 1, then the channel state at the tth time is expressed as:

[0120]

[0121] in, is the time-frequency deviation compensation factor of the nth group of APs at the jth time. The nth anchor AP shares the same compensation factor with the ith common AP in its group, that is, the APs in the same group use the same time-frequency deviation compensation factor. The index of the AP is defined as i. For the nth anchor AP, i = n; for the common AP, Then the estimated channel of the ith AP is It can be expressed as state s t Function:

[0122]

[0123] (2) Action

[0124] In Cell-free Massive MIMO Systems It is not easy to estimate, so the present invention proposes to use the CPU as a DDPG agent to estimate the time-frequency deviation of all anchor APs at the same time and compensate all AP channels. Define the time-frequency deviation phase factor of the nth group of APs at the tth time as Define the time-frequency deviation compensation factor of the nth anchor point AP at the tth time as According to formula (16), the time-frequency deviation compensation factor needs to satisfy At time t, the N-dimensional action for compensating the time-frequency deviation of each group of AP estimated channels is defined as:

[0125]

[0126] in,

[0127] When the agent gets the action After that, you need to set the channels of all APs Perform time-frequency offset compensation. The equivalent channel after time-frequency offset compensation on the i-th AP is expressed as in, represents the equivalent channel between the i-th AP and the k-th user, then action a is executed t After that, for the i-th AP, the channel in the next state is Right now

[0128]

[0129] (3) Rewards

[0130] The equivalent channel of the kth user at time t is expressed as After time-frequency deviation compensation, the state s t The SINR of the next k-th user is expressed as:

[0131]

[0132] The overall spectral efficiency is:

[0133]

[0134] The optimal compensation strategy can achieve SE (3) =SE (1) But due to It is impossible to estimate effectively, that is, it is impossible to directly use formula (31) to calculate Therefore, the present invention introduces an SINR perception method, and the SINRs measured by all users are reported to the AP and fed back to the CPU, so as to continuously improve the SINRs of all users, thereby improving the overall spectrum efficiency.

[0135] Defined in state s t The measured SINR of the kth user is expressed as The total spectrum efficiency based on the SINR perception algorithm is:

[0136]

[0137] Execute action a t-1 After that, the gain in total spectral efficiency is taken as the immediate reward r t (s t ,a t ),Right now

[0138]

[0139] The agent forms an interaction trajectory (s1, a1, r1, s2, a2, r2, ···, s T-1 ,a T-1 ,r T-1 ,s T ,···), state s t The cumulative rewards are:

[0140]

[0141] Among them, γ∈[0,1] is a discount factor, which is used to control the impact of future rewards on current rewards. The smaller the value, the smaller the current reward ratio, and vice versa, the long-term return is dominant.

[0142] Actor network is in state s t Execute action a t Afterwards, the CPU agent performs time-frequency deviation compensation on all AP channels and transfers the state to s t+1 At the same time, the user's SINR value is improved, and the CPU agent calculates the reward r based on the SINR measurement value reported by the user. tIn this process, the Actor network learns a deterministic action based on the current strategy and state, while the Critic network is responsible for evaluating the value of the action.

[0143] The DDPG algorithm converts the trained state, action, reward, and next state (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool In the subsequent learning process, the agent A batch of samples are randomly selected as the input of the network, and the Critic network and the Actor network are trained by the gradient descent method.

[0144] The DDPG algorithm flow chart is as follows Figure 2 shown.

[0145] The state space and action space in this scheme are multi-dimensional continuous spaces, and DDPG can effectively handle complex tasks with continuous state space and action space. First, DDPG uses deep neural networks to approximate policy functions and value functions, so it can flexibly handle high-dimensional continuous state and action spaces. Second, the Actor network in DDPG directly outputs deterministic actions instead of probability distributions, which simplifies the learning process and improves the stability of the strategy. Table 1 describes the AP synchronization algorithm flow based on DDPG:

[0146] Table 1

[0147]

[0148]

[0149] The concepts of episode and T are introduced in the DDPG algorithm to optimize the reinforcement learning process. An episode represents a complete learning cycle, which allows the agent to try different strategies, explore the environment and collect data in each cycle, while periodically resetting the environment to reduce strategy shocks and improve the stability of the algorithm. The agent needs to learn and adapt to the environment independently in each episode without relying on previous rewards. After each episode, the agent has the opportunity to re-evaluate the state of the environment and adjust the strategy.

[0150] In the present invention, the reward is a measure of the degree to which the action improves the overall spectral efficiency. Only after executing multiple actions and realizing multiple state transitions can the final optimal state be achieved. Therefore, T is introduced as the maximum number of steps for each episode, that is, the number of state transitions to reach the optimal state. If T is too large, the correlation of samples may increase, making the learning process unstable, and at the same time increasing the computational cost, causing the agent to overfit certain states and ignore other important states; if T is too small, the agent may not be able to fully explore the environment, resulting in the lack of generalization ability of the learned strategy, and may also result in sample waste, affecting learning efficiency. The reward R in each episode cycle e is cumulative, i.e. As the number of episodes increases, R e It will increase gradually until convergence.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. AP synchronization method based on deep reinforcement learning for non-cellular massive MIMO system, characterized by: The following steps are involved: S1: Establish a system model, including the number of APs, the number of users, the channel model, and the signal model; S2: construct the state space, including AP time-frequency deviation compensation factor and anchor AP estimation channel; S3: Constructing the action space, including the AP time-frequency deviation compensation factor; S4: define a reward function, taking the total spectrum efficiency gain as the reward; S5: The deep deterministic policy gradient algorithm DDPG is used to train the agent. The agent selects actions based on the current state and receives rewards from the environment to continuously learn the optimal time-frequency deviation compensation strategy. S6: Use the trained proxy model to adjust the AP time-frequency deviation compensation factor in real time to achieve synchronization between APs and improve the overall spectrum efficiency of the system.

2. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S1, the system model includes L single-antenna APs and K single-antenna users. The APs and users are connected via channels, and the channel model is a fast fading channel model.

3. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S2, the state space includes the accumulated compensation factors of the time-frequency deviations of N groups of APs and the estimated channel sets of N anchor APs.

4. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S3, the action space is an N-dimensional continuous space, representing the time-frequency deviation compensation factors of N groups of APs.

5. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S4, the reward function is the gain of the total spectral efficiency after executing the action.

6. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S5, the DDPG algorithm includes an Actor network, an Actor target network, a Critic network, and a Critic target network, and stores the experience data generated by the interaction between the agent and the environment through an experience replay pool.

7. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S6, the proxy model adjusts the AP time-frequency deviation compensation factor in real time, and calculates the total spectrum efficiency using SINR perception technology to optimize system performance.

8. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S6, the AP time-frequency deviation compensation factor satisfies: represents the time-frequency deviation compensation factor of the nth anchor point AP at the tth time; represents the conjugate of the time-frequency deviation of the nth anchor point AP at the current moment; At time t, the N-dimensional action for compensating the time-frequency deviation of each group of AP estimated channels is defined as: in, Represents the exponential value of the time-frequency deviation compensation factor of the nth anchor point AP at the tth time.

9. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S6, the equivalent channel after the AP channel time-frequency deviation compensation satisfies: When the agent gets the action After that, the channels of all APs Perform time-frequency deviation compensation; the equivalent channel after time-frequency deviation compensation on the i-th AP is expressed as in, represents the equivalent channel between the i-th AP and the k-th user, then action a is executed t After that, for the i-th AP, the channel in the next state is Right now Among them, s t (n) and a t (n) are the state and action of the nth AP at the tth moment.

10. The AP synchronization method based on deep reinforcement learning for a non-cellular massive MIMO system according to claim 1, characterized in that: In S6, the total spectrum efficiency based on the SINR perception algorithm is Defined in state s t The measured SINR of the kth user is expressed as The total spectrum efficiency based on the SINR perception algorithm is: Among them, τ p is the pilot length; τ c is the total signal length; K is the number of users; To obtain the expected operation.

Images