Multi-beam interference avoidance method and system based on multi-agent reinforcement learning

Abstract

The invention relates to a multi-beam interference avoidance method and system based on multi-agent reinforcement learning. The method comprises the following steps: S1, establishing an initial system model diagram according to a relationship between multiple beams generated by a satellite and a ground cell; s2, analyzing interference among multiple beams; s3, generating a service arrival model according to the ground state information; s4, constructing an M / G / 1 queuing model, and inputting an output result of the model in the S3 into the queuing model; s5, multi-beam context learning is carried out; s6, converting a beam interference avoidance problem into a multi-agent depth deterministic strategy gradient algorithm learning problem, and performing conversion definition of a Markov decision process problem; and S7, performing reinforcement learning of the MADDPG algorithm according to the conversion definition of the problem and the MADDPG algorithm, and planning an optimal strategy of satellite beam scheduling according to the maximum probability action value output by the plurality of beams of the satellite. According to the method, the problem that interference exists among multiple beams in an existing low-orbit satellite multi-beam system is solved, the method is more reasonable and flexible, interference can be effectively avoided, and the overall performance of the system is improved.

Description

Technical Field

[0001] The present invention belongs to the field of satellite communication technology, and specifically relates to a multi-beam interference avoidance method and system based on multi-agent reinforcement learning. Background Art

[0002] With the growing global demand for high-speed Internet and data transmission, satellite communication technology is also evolving. Multi-beam technology enables each beam to accurately cover a specific small area on the earth's surface, which not only meets the communication needs of high-density users, but also makes resource allocation more accurate. However, in multi-beam systems, the problem of beam interference seriously affects system performance. Beam interference between multiple beams refers to the mutual interference between beams when multiple beams exist at the same time, resulting in problems such as signal attenuation, increased bit error rate and decreased communication quality. Therefore, how to effectively suppress beam interference between multiple beams and improve the stability and reliability of the communication system has become an important direction of current research. Based on this, the present invention designs a multi-beam interference avoidance method and system based on multi-agent reinforcement learning, which reduces inter-beam interference while ensuring its fairness and improving system throughput. Summary of the invention

[0003] In view of the above problems, the present invention proposes a multi-beam interference avoidance method and system based on multi-agent reinforcement learning. The present invention analyzes the data packet flow in the cell using a queuing theory model to determine the optimal beam scheduling allocation scheme. The present invention solves the problems existing in the prior art, avoids interference in the existing low-orbit satellite multi-beam system, and improves the throughput within the beam while ensuring fairness between cells.

[0004] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A multi-beam interference avoidance method based on multi-agent reinforcement learning includes the following steps:

[0006] S1. Establish an initial system model diagram based on the multi-beams generated by the satellite and the relationship between the ground cells;

[0007] S2. Analyze the interference between multiple beams;

[0008] S3. Generate a service arrival model based on ground status information;

[0009] S4, construct the M / G / 1 vacation queue model, and input the output of the business arrival model in S3 into the M / G / 1 vacation queue model;

[0010] S5, perform multi-beam context learning;

[0011] S6. Transform the beam interference avoidance problem into a multi-agent deep deterministic policy gradient (MADDPG) algorithm learning problem, and perform the transformation definition of the Markov decision process (POMDP) ​​problem;

[0012] S7. According to the transformation definition of the problem and the MADDPG algorithm, reinforcement learning of the MADDPG algorithm is performed, and the optimal strategy for satellite beam scheduling is planned according to the maximum probability action value of multiple satellite beam outputs.

[0013] As a preferred solution, in step S1, an initial system model diagram is established based on the multi-beams generated by the satellite and the relationship between the ground cells, including:

[0014] The single satellite multi-beam model is adopted. The system includes low-orbit satellites, gateway stations and ground cells. The antenna uses a planar uniform phased array antenna array, which can randomly generate K beams, expressed as K = {k|k = 1, 2, ..., K}, to cover N cells on the ground, expressed as N = {n|n = 1, 2, ..., N}, where K <N。

[0015] As a preferred solution, in step S2, multi-beam interference analysis is performed, specifically including:

[0016] S21. According to step S1, when the same-frequency beam is performing link communication, the side lobe or main lobe of each spot beam overlaps with the main lobe of other beams in space. Therefore, in addition to the required signal, the main lobe or side lobe interference signal of the adjacent beam will also be received, thereby affecting the signal quality and data transmission rate of the receiving end.

[0017] S22. Based on S21, the carrier-to-interference ratio C / I of the terminal received signal is used to quantify the inter-beam interference:

[0018]

[0019] Among them, G t (θ0) Desired beam transmit antenna gain, G t,I (θ i ) is the antenna gain transmitted by other interference beams.

[0020] As a preferred solution, in step S3, generating a service arrival model according to the ground state information specifically includes:

[0021] Based on the geographical and topographical differences of ground cells, a service arrival model was established. The service volume is calculated by weighting the three factors of terrain, development and time. The geographical terrain is divided into four types: ocean, land, desert, and mountain. Different coefficients are assigned to each geographical terrain feature. The development status is divided into developing and developed. The time factor is assigned a value considering the 24-hour work and rest time of human life. The weight relationship of the three factors is considered in the calculation. At time t, the equipment deployment density of grid i is defined as:

[0022]

[0023] Among them, S i represents the area of ​​grid i, m represents the number of geographical environment types contained in the grid, and ρ j represents the device deployment density corresponding to geographical environment type j, which is calculated based on the weighted terrain and development conditions. i,j It represents the area occupied by geographical environment type j in grid i. The flow in the system follows a Poisson distribution with an arrival rate of λ.

[0024] As a preferred solution, in step S4, an M / G / 1 vacation queuing model is constructed, and the result output by the service arrival model in S3 is input into the queuing model, which specifically includes:

[0025] S41. The satellite can provide N queues to store the traffic arriving at each cell. The arrival traffic at time slot t is expressed as:

[0026] L t = {l t,n |n∈N}

[0027] Among them, l t,n is the arrival flow of cell n at time slot t, subject to arrival rate λ t,n The total traffic stored in the queue in time slot t is expressed as D t ={d t,n |n∈N}, where d t,n is the total flow stored in queue n in time slot t.

[0028] S42. Each cell is relatively independent. In the cell, data packets arrive in a Poisson process with an average arrival rate of λ. At the same time, after entering the cell, they need to queue up and wait for the beam to be processed, and follow the first-come-first-served principle. When the beam is turned on, the service time of each data packet is a random variable, obeying the general distribution, and the average service rate is μ on , the mean of the service time is E[S] and the variance is σ S 2 When the beam is turned off, the service rate is 0. At the same time, there are still data packets arriving at the cell and queuing in the turned-off state.

[0029] S43. Based on step S42, the data packet queuing waiting service process is designed as an M / G / 1 model, and the beam opening is regarded as a working state, and the beam closing is a vacation state.

[0030] S44. When the beam is turned on based on step S43, the average waiting delay (including queuing waiting time and service time) can be given by the Pollaczek-Khinchine (PK) formula:

[0031]

[0032] Among them, E[S 2 ] is the second moment of service time, is the utilization of the system.

[0033] During the beam-off period, the average beam-off time is assumed to be T off , then during the shutdown period, the average accumulated delay of each data packet is:

[0034] τ off =T off

[0035] S45: Based on step S44, the probability of beam opening is set to p on , the closing probability is p off , p on and p off It can be obtained by statistical analysis of the historical data of the cell. Taking into account the on and off states of the beam, the average waiting delay can be expressed as:

[0036] τ t,n =p on ·τ on +p off ·τ off

[0037] As a preferred solution, in step S4, the beam-on state is regarded as the working state, and the beam-off state is regarded as the vacation state; the information data obtained by analyzing the queuing model is the average waiting delay in the cell.

[0038] As a preferred solution, in step S5, the context learning of the multi-beams specifically includes:

[0039] S51. Use the operating status of all beams to form a context for interference avoidance. When a beam serves a cell, the beam is called an active state, otherwise it is called an idle state. The context vector of the operating status of all beams at observation time t is collected and defined as follows:

[0040] c(t)={c1(t),c2(t),…,c K(t)}

[0041] S52. The agent observes the state of the beam that interferes with its own transmission and applies the context, that is, each agent masks the state of the non-interfering beam to form its own version of the context. Based on the underlying deployment layout, the non-interfering beams of each beam can be easily identified and the corresponding mask settings can be applied. Each machine learning (ML) agent masks the state of the non-interfering beams in the context to form its own version of the context; the mask vector representing the non-interfering beam of beam bi can be defined as follows:

[0042] M i ={m 1|i ,m 2|i ,…,m K|i}

[0043] Among them, m k|i Indicates whether the activation of beam k will not interfere with beam i, that is, when beam k can interfere with beam i, m k|i becomes 1, otherwise becomes 0.

[0044] S53, c i (t) represents the observation context of beam i after masking all non-interfering beam states to 0:

[0045]

[0046] Where c(t)={c1(t),c2(t),…,c K (t)} is the total context observed by the agent initially.

[0047] As a preferred solution, in step S6, the beam interference avoidance problem is converted into a multi-agent deep deterministic policy gradient (MADDPG) algorithm learning problem, and the conversion definition of the Markov decision process (POMDP) ​​problem is performed, specifically including:

[0048] S61. Define the global state s including the average waiting delay τ of the ground cell i , the generated beam transmits the processed data packet q b And the data traffic D in the cell:

[0049] s={τ i ,q b ,D}

[0050] S62, define the context c of the t-slot beam observation and the basic observation feature s as the global state o:

[0051]

[0052] o={s,c}

[0053] S63. The agent should make decisions based on the state to improve long-term benefits. The agent dynamically adjusts the beam coverage of each time slot unit, that is, turns the beam on or off, and intelligently adjusts the selection based on the state in each time slot. Therefore, the action performed at time t is defined as:

[0054] a={x1,x1,…,x k |x i ∈{0,1}}

[0055] S64. To maximize the defined long-term optimization goal, use data throughput and latency fairness as immediate rewards:

[0056] r=αTP total -(1-α)TD

[0057] Among them, TP total represents throughput, and TD represents delay difference.

[0058] As a preferred solution, in step S7, according to the transformation definition of the problem and the MADDPG algorithm, the reinforcement learning of the MADDPG algorithm is performed, and the optimal strategy for satellite beam scheduling is planned according to the maximum probability action value output by multiple satellite beams, including:

[0059] S71. Randomly initialize the actor networks and critic networks of all beams and configure the experience replay buffer.

[0060] S72. Use the deterministic strategy network to collect observation data of the current environment. The data includes the beam context c of time slot t and the basic observation feature s, where s includes the average waiting delay of the ground cell, the number of data packets processed by the current beam, the number of data packets discarded in the cell, and the data traffic in the cell.

[0061] o={s,c}

[0062] S73. Based on the environmental observation data obtained in step S72, determine the target beam forming action of the low-orbit satellite and execute the action.

[0063] S74. After completing the beam decision action, record the reward obtained and the environmental observation data of the next state, and store the current environmental observation data, the execution action of the target satellite, the reward obtained and the new environmental observation data in the experience playback buffer.

[0064] S75. Randomly extract a batch of experiences from the experience replay buffer for training, update the parameters of the actor network and the critic network, aiming to maximize the Q value of the critic network, minimize the error of the Q value, and use the updated Q network to generate the optimal policy path.

[0065] S76. Repeat steps S72-S75 until the algorithm converges (for example, the beam scheduling strategy becomes stable during the training process and no longer changes significantly), thereby determining the optimal beam scheduling strategy.

[0066] The present invention also discloses a multi-beam interference avoidance system based on multi-agent reinforcement learning, which is used to execute the above method and includes the following modules:

[0067] Initial multi-beam system model diagram construction module: establishes the initial system model diagram based on the multi-beams generated by the satellite and the relationship between the ground cells;

[0068] Information data acquisition module: Analyze the interference between multiple beams and quantify the interference according to the multi-beam characteristics;

[0069] Business arrival model module: generates business arrival model based on ground status information;

[0070] Queuing model module: constructs an M / G / 1 queuing model, inputs service arrival information into the queuing model, and generates an independent queuing model for each cell;

[0071] Context learning module: Multi-beam context learning, generating unique context for each beam based on the underlying beam information;

[0072] Conversion module: converts the beam interference avoidance problem into a multi-agent deep deterministic policy gradient algorithm learning problem and performs a conversion definition of the Markov decision process problem;

[0073] Reinforcement learning and beam scheduling module: Based on the transformation definition of the problem and the multi-agent deep deterministic policy gradient algorithm, reinforcement learning of the multi-agent deep deterministic policy gradient algorithm is performed, and the optimal strategy for satellite beam scheduling is planned based on the maximum probability action value output by multiple satellite beams.

[0074] Compared with the prior art, the present invention has the following technical effects:

[0075] (1) The present invention adopts a context learning model. The target beam learns its own context environment and masks the non-interference beam through the underlying beam information, and finally generates specific context state information for each target beam. This model can effectively capture and utilize the context information of the environment while saving state space, thereby optimizing the beam scheduling strategy and improving the overall performance and adaptability of the system.

[0076] (2) The present invention selects the multi-agent deep deterministic policy gradient (MADDPG) algorithm in reinforcement learning. The observed environment state information and context information are simultaneously input into some Markov decision formulas, which can better adapt to the changes and complexity of the environment. At the same time, the algorithm can handle complex decision-making problems in a multi-agent environment, and improve the overall effect of the system by coordinating the behaviors of multiple agents, thereby achieving more efficient beam scheduling and interference management. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a flow chart of a multi-beam interference avoidance method based on multi-agent reinforcement learning in a preferred embodiment of the present invention;

[0078] Figure 2 A model diagram based on context learning according to a preferred embodiment of the present invention;

[0079] Figure 3 This is a model diagram of a multi-agent deep deterministic policy gradient algorithm learning involved in a preferred embodiment of the present invention.

[0080] Figure 4 This is a block diagram of a multi-beam interference avoidance system based on multi-agent reinforcement learning according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0082] like Figure 1As shown, this embodiment proposes a multi-beam interference avoidance method based on multi-agent reinforcement learning, which combines the multi-agent deep deterministic policy gradient algorithm model and the context learning model, and uses a queuing model. First, an initial system model diagram is established based on the multi-beams generated by the satellite and the relationship between the ground cells, and the multi-beam interference analysis is performed. The service arrival model is generated in combination with the ground state information, and the model output result is input into the queuing model to construct an M / G / 1 queuing model. The beam interference avoidance problem is converted into a multi-agent deep deterministic policy gradient (MADDPG) algorithm learning problem, and the transformation definition of the Markov decision process (POMDP) ​​problem and the context learning of the multi-beam are performed. According to the transformation definition of the problem and the MADDPG algorithm, the reinforcement learning of the MADDPG algorithm is performed, and the optimal strategy for satellite beam scheduling is planned according to the maximum probability action value output by multiple satellite beams. The specific steps of this embodiment are as follows:

[0083] S1. Establish an initial system model diagram based on the multi-beams generated by the satellite and the relationship between the ground cells;

[0084] S2. Analyze the interference between multiple beams;

[0085] S3. Generate a service arrival model based on ground status information;

[0086] S4, build the M / G / 1 vacation queuing model, and input the output of the business arrival model in S3 into the queuing model;

[0087] S5, perform multi-beam context learning;

[0088] S6. Transform the beam interference avoidance problem into a multi-agent deep deterministic policy gradient (MADDPG) algorithm learning problem, and perform the transformation definition of the Markov decision process (POMDP) ​​problem;

[0089] S7. According to the transformation definition of the problem and the MADDPG algorithm, reinforcement learning of the MADDPG algorithm is performed, and the optimal strategy for satellite beam scheduling is planned according to the maximum probability action value of multiple satellite beam outputs.

[0090] The following is a detailed description of each step of this embodiment.

[0091] In step S1, an initial system model diagram is established based on the multi-beams generated by the satellite and the relationship between the ground cells, including:

[0092] The single satellite multi-beam model is adopted. The system includes low-orbit satellites, gateway stations and ground cells. The antenna uses a planar uniform phased array antenna array, which can randomly generate K beams, expressed as K = {k|k = 1, 2, ..., K}, to cover N cells on the ground, expressed as N = {n|n = 1, 2, ..., N}, where K <N。

[0093] In step S2, multi-beam interference analysis is performed, specifically including:

[0094] S21. According to step S1, when the same-frequency beam is performing link communication, the side lobe or main lobe of each spot beam overlaps with the main lobe of other beams in space. Therefore, in addition to the required signal, the main lobe or side lobe interference signal of the adjacent beam will also be received, thereby affecting the signal quality and data transmission rate of the receiving end.

[0095] S22. Based on S21, the carrier-to-interference ratio C / I of the terminal received signal is used to quantify the inter-beam interference:

[0096]

[0097] Among them, G t (θ0) Desired beam transmit antenna gain, G t,I (θ i ) is the antenna gain transmitted by other interference beams.

[0098] In step S3, a service arrival model is generated according to the ground state information, specifically including:

[0099] Based on the geographical and topographical differences of ground cells, a service arrival model is established. The service volume is calculated by weighting three factors: topography, development, and time. At time t, the device deployment density of grid i is defined as:

[0100]

[0101] Among them, S i represents the area of ​​grid i, m represents the number of geographical environment types contained in the grid, and ρ j represents the device deployment density corresponding to geographical environment type j, which is calculated based on the weighted terrain and development conditions. i,j It represents the area occupied by geographical environment type j in grid i. The flow in the system follows a Poisson distribution with an arrival rate of λ.

[0102] In step S4, an M / G / 1 vacation queuing model is constructed, and the result output by the service arrival model in S3 is input into the queuing model, which specifically includes:

[0103] S41. The satellite can provide N queues to store the traffic arriving at each cell. The arrival traffic at time slot t is expressed as:

[0104] L t = {l t,n |n∈N}

[0105] Among them, l t,n is the arrival flow of cell n at time slot t, subject to arrival rate λ t,n The total traffic stored in the queue in time slot t is expressed as D t ={d t,n |n∈N}, where d t,n is the total flow stored in queue n in time slot t.

[0106] S42. Each cell is relatively independent. In the cell, data packets arrive in a Poisson process with an average arrival rate of λ. At the same time, after entering the cell, they need to queue up and wait for the beam to be processed, and follow the first-come-first-served principle. When the beam is turned on, the service time of each data packet is a random variable, obeying the general distribution, and the average service rate is μ on , the mean of the service time is E[S] and the variance is σ S 2 When the beam is turned off, the service rate is 0. At the same time, there are still data packets arriving at the cell and queuing in the turned-off state.

[0107] S43. Based on step S42, the data packet queuing waiting service process is designed as an M / G / 1 model, and the beam opening is regarded as a working state, and the beam closing is a vacation state.

[0108] S44. When the beam is turned on based on step S43, the average waiting delay (including queuing waiting time and service time) can be given by the Pollaczek-Khinchine (PK) formula:

[0109]

[0110] Among them, E[S 2 ] is the second moment of service time, is the utilization of the system.

[0111] During the beam-off period, the average beam-off time is assumed to be T off , then during the shutdown period, the average accumulated delay of each data packet is:

[0112] τ off =T off

[0113] S45: Based on step S44, the probability of beam opening is set to p on , the closing probability is p off , p on and poff It can be obtained by statistical analysis of the historical data of the cell. Taking into account the on and off states of the beam, the average waiting delay can be expressed as:

[0114] τ t,n =p on ·τ on +p off ·τ off

[0115] This embodiment provides a multi-beam interference avoidance method based on multi-agent reinforcement learning, using a contextual learning model. The model diagram is referenced Figure 2 As shown, by using this model to learn and analyze the information of each beam, the interference beam can be identified in advance and the calculation complexity can be reduced. The specific steps include:

[0116] S51: Use the operating status of all beams to form a context to avoid interference. Specifically:

[0117] S511. The operating state of a beam can be an active state or an idle state. When the beam serves a cell, it is called an active state, otherwise it is called an idle state.

[0118] S512: Set a separate agent for each beam to independently observe the context and interact with the environment. Collect the context vectors of all beam operation states at observation time t, defined as follows:

[0119] c(t)={c1(t),c2(t),…,c K (t)}

[0120] S52. Each ML agent masks the state of non-interfering beams in the context to form its own version of the context.

[0121] Based on the underlying deployment layout, the non-interfering beams for each beam can be easily identified and the corresponding mask settings can be applied. The mask vector representing the non-interfering beams for beam i can be defined as follows

[0122] M i ={m 1|i ,m 2|i ,…,m K|i}

[0123] Among them, m k|i Indicates whether the activation of beam k will not interfere with beam i, that is, when beam k can interfere with beam i, m k|i becomes 1, otherwise becomes 0.

[0124] S53, based on all beam state information obtained in step S51 and the mask information based on the non-interference beam in step S52, obtain the context information of each agent. i (t) represents the observation context of beam i after masking all non-interfering beam states to 0. i (t) can be expressed as follows.

[0125]

[0126] This embodiment is a multi-beam interference avoidance method based on multi-agent reinforcement learning, based on a multi-agent deep deterministic policy gradient algorithm model. The model diagram is referenced Figure 3 As shown in the figure, in this network, the Q network receives the observation values ​​of all beams generated by the satellite and the environmental information composed of the state information generated by context learning and the actions of all intelligent agents as input during the training phase, so as to centrally calculate the action-value function of the beam. Each beam will learn a Q value separately as feedback for the beam action. The Q network is trained based on the estimated Q value and the actual Q value, and the satellite updates the strategy based on the feedback from the Q network. When each beam is fully trained, each beam can take appropriate actions according to its state, and the state or action of other beams is not required at this time. In the execution phase, the local observation of the beam is used as input and its action is output. It specifically includes the following steps:

[0127] S61. Define the basic observation characteristics s of the environment, including the average waiting delay τ of the ground cell i , the satellite generates a beam to transmit the processed data packet q b And the data traffic D in the cell:

[0128] s={τ i ,q b ,D}

[0129] S62, define the context c of the t-slot beam observation and the basic observation feature s as the global state o:

[0130]

[0131] o={s,c}

[0132] S63, define the action space as beam opening or closing:

[0133] a={x1,x2,…,x k |x i ∈{0,1}}

[0134] Among them, x i =0 means beam is off, x i =1 means turning on the beam.

[0135] S64. Define a reward function based on the characteristics of intra-cell traffic and waiting delay:

[0136] r=αTP total -(1-α)TD

[0137] Among them, TP total represents throughput, and TD represents delay difference.

[0138] Step S7 is specifically as follows:

[0139] S71. Randomly initialize the actor networks and critic networks of all beams and configure the experience replay buffer.

[0140] S72. Use the deterministic strategy network to collect observation data of the current environment. The data includes the beam context c of time slot t and the basic observation feature s, where s includes the average waiting delay of the ground cell, the number of data packets processed by the current beam, the number of data packets discarded in the cell, and the data traffic in the cell.

[0141] S73. Based on the environmental observation data obtained in step S72, determine the target beam forming action of the low-orbit satellite and execute the action.

[0142] S74. After completing the beam decision action, record the reward obtained and the environmental observation data of the next state, and store the current environmental observation data, the execution action of the target satellite, the reward obtained and the new environmental observation data in the experience playback buffer.

[0143] S75. Randomly extract a batch of experiences from the experience replay buffer for training, update the parameters of the actor network and the critic network, aiming to maximize the Q value of the critic network, minimize the error of the Q value, and use the updated Q network to generate the optimal policy path.

[0144] S76. Repeat steps S71-S75 until the algorithm converges, thereby determining the optimal beam scheduling strategy.

[0145] The present invention solves the problem of interference between multiple beams in the existing low-orbit satellite multi-beam system, is more reasonable and flexible, can effectively avoid interference, and improves the overall performance of the system.

[0146] like Figure 4 As shown, this embodiment is a multi-beam interference avoidance system based on multi-agent reinforcement learning, which is used to execute the method of the above embodiment, and includes the following modules:

[0147] Initial multi-beam system model diagram construction module: establishes the initial system model diagram based on the multi-beams generated by the satellite and the relationship between the ground cells;

[0148] Information data acquisition module: According to the multi-beam characteristics of the system, analyze the interference between multiple beams and quantify the interference;

[0149] Business arrival model module: generates business arrival model based on ground status information;

[0150] Queuing model module: constructs the M / G / 1 vacation queuing model, inputs the output of the service arrival model into the queuing model, and generates an independent queuing model for each cell;

[0151] Context learning module: Multi-beam context learning, generating unique context for each beam based on the underlying beam information;

[0152] Conversion module: converts the beam interference avoidance problem into a multi-agent deep deterministic policy gradient algorithm learning problem and performs a conversion definition of the Markov decision process problem;

[0153] Reinforcement learning and beam scheduling module: Based on the transformation definition of the problem and the multi-agent deep deterministic policy gradient algorithm, reinforcement learning of the multi-agent deep deterministic policy gradient algorithm is performed, and the optimal strategy for satellite beam scheduling is planned based on the maximum probability action value output by multiple satellite beams.

[0154] The above description is only a detailed description of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, according to the ideas provided by the present invention, there will be changes in the specific implementation methods, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. A multi-beam interference avoidance method based on multi-agent reinforcement learning, characterized by The following steps are involved: S1. Establish an initial system model diagram based on the multi-beams generated by the satellite and the relationship between the ground cells; S2. Analyze the interference between multiple beams; S3. Generate a service arrival model based on ground status information; S4, construct the M / G / 1 vacation queue model, and input the output of the business arrival model in S3 into the M / G / 1 vacation queue model; S5, perform multi-beam context learning; S6. Transform the beam interference avoidance problem into a multi-agent deep deterministic policy gradient algorithm learning problem, and perform a transformation definition of the Markov decision process problem; S7. According to the transformation definition of the problem and the multi-agent deep deterministic policy gradient algorithm, reinforcement learning of the multi-agent deep deterministic policy gradient algorithm is carried out, and the optimal strategy for satellite beam scheduling is planned according to the maximum probability action value output by multiple satellite beams.

2. According to claim 1, a multi-beam interference avoidance method based on multi-agent reinforcement learning is characterized in that: In step S1, a single satellite multi-beam model is adopted, including a low-orbit satellite, a gateway station and a ground cell; the antenna selection sets a planar uniform phased array antenna array, which can randomly generate K beams, represented by K = {k|k = 1, 2, ..., K}, covering N ground cells, represented by N = {n|n = 1, 2, ..., N}, where K <N。 3. According to claim 1, a multi-beam interference avoidance method based on multi-agent reinforcement learning is characterized in that: In step S2, the carrier-to-interference ratio C / I of the terminal received signal is used to quantify the inter-beam interference: Among them, G t (θ0) desired beam transmit antenna gain, G t,I (θ i ) is the antenna gain transmitted by other interference beams.

4. According to claim 1, a multi-beam interference avoidance method based on multi-agent reinforcement learning is characterized in that: In step S3, considering the geographical and topographical differences of the ground cells, a service arrival model is established. The model comprehensively reflects the service volume under different conditions by weighted calculation of the topographical characteristics, development status and time factors. At time t, the equipment deployment density of grid i is defined as: Among them, S i represents the area of ​​grid i, m represents the number of geographical environment types contained in the grid, and ρ j represents the device deployment density corresponding to geographical environment type j, S i,j Represents the area occupied by geographical environment type j in grid i; the flow follows a Poisson distribution with an arrival rate of λ.

5. According to claim 4, a multi-beam interference avoidance method based on multi-agent reinforcement learning is characterized in that: In step S4, the M / G / 1 vacation queuing model is constructed, and the result of the business arrival model output in S3 is input into the queuing model, which specifically includes: S41. The satellite provides N queues for storing the traffic arriving at each cell. The arrival traffic at time slot t is expressed as: L t ={l t,n |n∈N} Among them, l t,n is the arrival flow of cell n at time slot t, subject to arrival rate λ t,n Poisson distribution; the total flow stored in the queue in time slot t is expressed as D t ={d t,n |n∈N}, where d t,n is the total flow stored in queue n in time slot t; S42. Each cell is relatively independent. In the cell, data packets arrive in a Poisson process with an average arrival rate of λ. At the same time, after entering the cell, they need to queue up and wait for the beam to be processed, and follow the first-come-first-served principle. When the beam is turned on, the service time of each data packet is a random variable, obeying the general distribution, and the average service rate is μ. on , the mean of the service time is E[S] and the variance is σ S 2 ;When the beam is turned off, the service rate is 0, and at the same time, data packets still arrive at the cell and are queued in the turned-off state; S43, the data packet queuing waiting service process is designed as an M / G / 1 vacation queuing model, and the beam opening is regarded as the working state, and the beam closing is the vacation state; When S44 and beam are turned on, the average waiting delay is given by the PK formula: Among them, E[S 2 ] is the second moment of service time, is the utilization of the system; During the beam closing period, the average beam closing time is T off , then during the shutdown period, the average accumulated delay of each data packet is: t off =T off S45. Let the probability of beam opening be p on , the closing probability is p off , p on and p off The average waiting delay is obtained by statistically analyzing the historical data of the cell and taking into account the on and off states of the beam: t t,n =p on ·t on +p off ·t off 。 6. According to claim 5, a multi-beam interference avoidance method based on multi-agent reinforcement learning is characterized in that: Step S5 specifically includes: S51. Use the operating status of all beams to form a context for interference avoidance. When a beam serves a cell, the beam is called an active state, otherwise it is called an idle state. The context vector of the operating status of all beams at observation time t is collected and defined as follows: c(t)={c1(t),c2(t),…,c K (t)} Among them, c K (t) is the context vector information of beam K collected at observation time t; S52. Each machine learning agent masks the state of the non-interfering beams in the context to form its own version of the context; the mask vector representing the non-interfering beam of beam i is defined as follows: M i ={m 1|i ,m 2|i ,…,m K|i } Among them, m k|i Indicates whether the activation of beam k will not interfere with beam i. When beam k can interfere with beam i, m k|i becomes 1, otherwise becomes 0; S53, based on the operating status information of all beams obtained in step S51 and the mask vector information based on the non-interference beams in step S52, the context information of each agent is obtained; c. i (t) is expressed as follows:

7. The multi-beam interference avoidance method based on multi-agent reinforcement learning according to claim 6 is characterized in that: Step S6 specifically includes: S61. Define the basic observation characteristics s of the environment, including the average waiting delay τ of the ground cell i , the satellite generates a beam to transmit the processed data packet q b And the data traffic D in the cell: s={τ i ,q b ,D} S62, define the context c of the t-slot beam observation and the basic observation feature s as the global state o: o={s,c} S63, define the action space as beam opening or closing: a={x1,x2,…,x k |x i ∈{0,1}} When x i =0 means beam is off, x i =1 means turning on the beam; S64. Define a reward function based on the characteristics of intra-cell traffic and waiting delay: r=αTP total -(1-a)TD Among them, α is the weight coefficient, TP total represents throughput, and TD represents delay difference.

8. A multi-beam interference avoidance method based on multi-agent reinforcement learning according to any one of claims 1 to 7, characterized in that: Step S7 specifically includes: S71. Randomly initialize the actor network and critic network of all beams, and configure the experience replay buffer; S72. Using a deterministic strategy network to collect observation data of the current environment, the data includes the beam context c of time slot t and the basic observation feature s, where s includes the average waiting delay of the ground cell, the number of packets processed by the current beam, the number of packets discarded in the cell, and the data traffic in the cell; S73. Based on the environmental observation data obtained in step S72, determine the target beam forming action of the low-orbit satellite and execute the action; S74. After executing the beam decision action, record the reward obtained and the environmental observation data of the next state, and store the current environmental observation data, the execution action of the target satellite, the reward obtained and the new environmental observation data in the experience playback buffer; S75. Randomly extract a batch of experiences from the experience replay buffer for training, update the parameters of the actor network and the critic network, aiming to maximize the Q value of the critic network, minimize the error of the Q value, and use the updated Q network to generate the optimal policy path; S76. Repeat steps S71-S75 until convergence, thereby determining the optimal beam scheduling strategy.

9. A multi-beam interference avoidance system based on multi-agent reinforcement learning, used to execute the method according to any one of claims 1 to 8, characterized in that: Includes the following modules: Initial multi-beam system model diagram construction module: establishes the initial system model diagram based on the multi-beams generated by the satellite and the relationship between the ground cells; Information data acquisition module: Analyze the interference between multiple beams and quantify the interference according to the multi-beam characteristics; Business arrival model module: generates business arrival model based on ground status information; Queuing model module: constructs the M / G / 1 vacation queuing model, inputs the output of the service arrival model into the M / G / 1 vacation queuing model, and generates an independent queuing model for each cell; Context learning module: Multi-beam context learning, generating unique context for each beam based on the underlying beam information; Conversion module: converts the beam interference avoidance problem into a multi-agent deep deterministic policy gradient algorithm learning problem and performs a conversion definition of the Markov decision process problem; Reinforcement learning and beam scheduling module: Based on the transformation definition of the problem and the multi-agent deep deterministic policy gradient algorithm, reinforcement learning of the multi-agent deep deterministic policy gradient algorithm is performed, and the optimal strategy for satellite beam scheduling is planned based on the maximum probability action value output by multiple satellite beams.

Images