A MIMO mode and modulation selection method based on deep reinforcement learning

Abstract

The present invention discloses a MIMO mode and modulation selection method based on deep reinforcement learning in the field of information and communication technology, aiming to solve the problem that the existing technology will cause a sharp decline in MIMO transmission performance in a non-stationary environment. A non-stationary MIMO channel model is established; according to the non-stationary MIMO channel model, the problem of jointly optimizing the MIMO mode and modulation mode is modeled as a Markov decision process, and the state space, action space, transition probability and reward function are defined; the PPO deep reinforcement learning algorithm is used to solve the Markov decision process to obtain the optimal MIMO mode and modulation mode. The present invention can effectively solve problems such as step size and learning rate, ensure the convergence of the algorithm, and use deep reinforcement learning neural networks to design a joint transmission strategy suitable for the current MIMO channel environment. It can ensure transmission reliability and maximize spectrum efficiency under the transmission conditions of non-stationary channels.

Description

Technical Field

[0001] The present invention relates to the field of information and communication technology, and in particular to a MIMO mode and modulation method selection method based on deep reinforcement learning. Background Art

[0002] In recent years, with the increasingly stringent and urgent requirements for information transmission, the reliability and effectiveness of existing communication systems are facing severe challenges. Traditional single-antenna technology cannot effectively solve problems such as low transmission rate, low spectrum efficiency and low energy efficiency. Multiple-input-multi-out (MIMO) technology integrates multiple antennas at the transmitting or receiving end and uses spatial domain resources to provide three types of gains for wireless communication systems, namely beamforming gain, multiplexing gain and diversity gain. The above gains are obtained through beamforming (BF), spatial multiplexing (SM) and spatial diversity (SD) technologies respectively.

[0003] BF technology can concentrate energy in the transmission direction and improve the signal-to-noise ratio at the receiving end. It can not only improve the link quality and communication coverage, but also serve as an important means of anti-interference and anti-eavesdropping. SM technology can provide higher channel capacity and spectrum efficiency, while SD technology is used to improve the reliability of transmission. It is worth noting that the above three gains cannot be obtained at the same time. This is because the three MIMO technologies are not only different in technical principles, but also in their respective applicable propagation environments. These three potential gains are highly dependent on channel characteristics. BF technology can only work in a channel environment dominated by the Light-of-sight (LoS) component, which relies on the high spatial correlation of MIMO subchannels. If the LoS component is the main component, the received signal strength between different antennas is approximately equal, and the spatial degree of freedom is 1. By adjusting the transmitting phase, the power is concentrated in the expected transmission direction to obtain power gain. When there are multiple scatterers between the transceivers, the LoS component is blocked by obstacles or the influence of the NLoS component dominates, the subchannel space is relatively independent and beamforming cannot be performed. However, at this time, the channel degree of freedom is high, and SD or SM mode can be used to improve link reliability and transmission rate. When the reliability requirement is higher than the transmission rate requirement, in order to ensure the transmission reliability of the system under poor transmission conditions (low signal-to-noise ratio), a fixed SD mode with a high diversity order is generally used to combat deep fading. The receiving end receives the same information from different paths and can obtain multiple copies of independently faded data symbols. The signal quality is enhanced through signal processing, thereby reducing the bit error rate. However, the large amount of redundant information in the SD mode greatly reduces the frequency efficiency. Therefore, when the channel conditions are good (high signal-to-noise ratio), the SM mode can be selected to use the independence of spatial subchannels to transmit different information in parallel on the same spectrum, thereby effectively increasing the data transmission rate and thus improving the spectrum utilization. In addition, the transmission modulation mode has an important impact on the spectrum efficiency and error performance of the MIMO system. When the channel conditions are good, high-order transmission is suitable to improve the spectrum efficiency. When the channel conditions are poor, only low-order transmission can be used. Fixed MIMO transmission mode and modulation mode will cause a sharp decline in MIMO transmission performance in a non-stationary environment.

[0004] In a non-stationary channel environment, affected by complex transmission environment, obstacle shielding, and dynamic mobility, the MIMO channel characteristics are dynamically unknown. Fixed MIMO transmission strategies will cause gain loss. Most existing MIMO technologies focus on improving single gain, while ignoring the impact of its non-stationary channel environment on the performance of MIMO technology. The traditional fixed single MIMO mode and modulation method are no longer applicable to time-varying and non-stationary wireless channels, and it is necessary to explore an efficient, intelligent and adaptive transmission strategy. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a MIMO mode and modulation selection method based on deep reinforcement learning, which can ensure transmission reliability and maximize spectrum efficiency under non-stationary channel transmission conditions, and has practical significance and good application prospects.

[0006] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0007] On the one hand, the present invention provides a MIMO mode and modulation mode selection method based on deep reinforcement learning, comprising:

[0008] Establish a non-stationary MIMO channel model;

[0009] According to the non-stationary MIMO channel model, the problem of jointly optimizing the MIMO mode and modulation mode is modeled as a Markov decision process, and the state space, action space, transition probability and reward function are defined therein;

[0010] The PPO deep reinforcement learning algorithm is used to solve the Markov decision process to obtain the optimal MIMO mode and modulation method;

[0011] The expression of the joint optimization of MIMO mode and modulation method after Markov decision process modeling is as follows:

[0012]

[0013] Where T represents the entire communication transmission time, 0<γ<1, represents the discount factor, which is used to describe the importance of current and future rewards, and r t (s t ,a t ) indicates that the agent is in state s t When , execute action a t Finally, the reward is obtained after interacting with the environment.

[0014] Optionally, the expression of the non-stationary channel model H is as follows:

[0015]

[0016] Where PL is the free space loss, ξ(t) is the Rice factor, z0(t) and z q (t) all represent additive Gaussian white noise, which obeys represents the noise variance Indicates dimension N r ×N r The unit diagonal matrix of represents the sight distance component matrix, θ Drepresents the horizontal angle of AOD, represents the vertical angle of AOD, and represents the antenna steering vector at the transmitting and receiving ends, θ A represents the horizontal angle of AOA, represents the vertical angle of AOA, Q represents the total number of non-line-of-sight components, represents the non-line-of-sight component in the Rice fading channel, each element of which obeys

[0017] Optionally, the state space In the example, the state of the tth signal frame is in, represents the spatial correlation coefficient of the channel model, and SNR(t) represents the estimated signal-to-noise ratio of the channel model;

[0018] The action space In the example, the action of the tth signal frame is a t ={k(t),m(t)}, where Indicates MIMO transmission mode, Indicates the modulation mode, action space The size is

[0019] The transition probability Used to describe the changes in the non-stationary channel environment. t Next, perform action a t After that, in s t+1 The transition probability is

[0020] The reward function The expression is as follows:

[0021]

[0022] Among them, r t (s t ,a t ) indicates that the agent is in state s t When , execute action a t After that, the reward obtained by feedback after interacting with the environment, η(t) represents the spectral efficiency of the t-th signal frame, r1 represents the positive reward, r2 represents the penalty, P e represents the bit error rate, P e,th Indicates the preset bit error rate threshold.

[0023] Optionally, the spatial correlation coefficient of the channel model The expression is as follows:

[0024]

[0025] Among them, N t Indicates the number of antennas at the transmitting end, N r Indicates the number of antennas at the receiving end, represents the spatial correlation of the transmitting antenna, Indicates the spatial correlation of the receiving end, diag m [·] and diag n [·] indicates that the m-th and n-th diagonals of the channel model are vectorized. m [·]} l and {diag n [·]} l represents the lth element in the vector obtained after vectorization, H represents the non-stationary channel model, and H H represents the conjugate transpose of the matrix H.

[0026] Optionally, solving the Markov decision process using the PPO deep reinforcement learning algorithm includes the following steps:

[0027] Initialize the neural network parameters θ a,new ,θ a,old ,θ c , set the experience playback unit capacity to N E , the number of cycles e = 1, the maximum number of cycles E max ;

[0028] Let θ a,old =θ a,new ;

[0029] Initialize state s0;

[0030] Let the number of iterations t = 1, and set the maximum number of iterations T max ;

[0031] Observe the environmental status t , and the environmental state s t Input the actor-critic network so that the critic network outputs the value function V π (s t θ c ), the actor network outputs the action selection strategy π(a|s t θ a,old );

[0032] According to the action selection strategy π(a|s t θ a,old ), execute action a t , and transfer to the next state s t+1 ;

[0033] Evaluate the bit error rate performance and spectral efficiency, and calculate the reward function r t ;

[0034] The tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience playback unit, and the following judgment is performed: If Mod(t+1,B m )==0, then sample the historical data of Mini-batch from the experience playback unit to calculate the advantage function, and update the network parameter θ by gradient descent a,new , let θ a,old =θ a,new ; Otherwise, the network parameters are not updated;

[0035] Let t=t+1, if t≤T max , then the observed environment state s is returned t Step, execute the next iteration, otherwise end;

[0036] Let e=e+1, if e≤E max , then return to the step of initializing state s0 and execute the next iteration, otherwise end.

[0037] Optionally, the advantage function is used to measure the quality of an action, and its expression is as follows:

[0038]

[0039] Among them, GAE(γ,λ) represents the ubiquitous advantage estimation, 0<γ<1, represents the discount factor, λ∈[0,1], represents the GA factor, which is used to control the size of the bias and variance. r t Indicates that the agent is in state s t When , execute action a t After that, the reward obtained by interacting with the environment is V π (s t θ c ) indicates that the agent is in state s t The value function when V π (s t+1 θ c ) indicates that the agent is in state s t+1 The value function when

[0040] when , it means that the current action is better, otherwise, it is considered that the current action cannot obtain a good reward function.

[0041] Optionally, the network parameter θa,new The expression for gradient descent update is as follows:

[0042]

[0043] Among them, π θ (τ) represents the action selection strategy, ρ t (θ a ) represents the magnitude of the strategy update, represents the advantage function, clip(·) represents the clip function, ε clip represents the hyperparameter, clip(ρ t (θ a ),1-ε clip ,1+ε clip ) represents the t (θ a ) is limited to (1-ε clip ,1+ε clip ).

[0044] In a second aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning as described in any one of the first aspects are implemented.

[0045] In a third aspect, the present invention provides a computer device / equipment / system, comprising:

[0046] Memory, for storing computer programs / instructions;

[0047] A processor is used to execute the computer program / instructions to implement the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning as described in any one of the first aspects.

[0048] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning described in any one of the first aspects are implemented.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The MIMO mode and modulation selection method based on deep reinforcement learning provided by the present invention performs a joint optimization scheme for the MIMO transmission mode and modulation mode, considers the transmission conditions of non-stationary channels, and maximizes the spectrum efficiency while ensuring the transmission reliability; at the same time, the selection of three transmission modes of BF, SD, and SM is considered, and the modulation mode is optimized to ensure the performance of MIMO technology in a non-stationary transmission environment;

[0051] 2. The computer-readable storage medium, computer device / equipment / system and computer program product provided by the present invention can execute the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning provided by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a MIMO mode and modulation method selection method based on deep reinforcement learning according to an embodiment of the present invention;

[0053] Figure 2 A model diagram of a MIMO transmission system provided according to an embodiment of the present invention;

[0054] Figure 3 A two-state birth and death process transition diagram provided according to an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a PPO network provided according to an embodiment of the present invention;

[0056] Figure 5 A performance comparison diagram provided according to an embodiment of the present invention;

[0057] Figure 6 This is a diagram showing the effects of different transmit powers provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0059] It should be noted that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0060] The present invention proposes a MIMO mode and modulation selection method based on deep reinforcement learning, which can effectively solve problems such as step size and learning rate, thereby ensuring the convergence of the algorithm. A neural network is used to extract, analyze and infer features of a complex MIMO channel environment, and the corresponding multi-dimensional communication parameters are matched according to the obtained channel characteristics. A joint transmission strategy suitable for the current MIMO channel environment is designed, which can ensure transmission reliability and maximize spectrum efficiency under the transmission conditions of non-stationary channels.

[0061] Embodiment 1:

[0062] The embodiment of the present invention discloses a method for selecting a MIMO mode and a modulation method based on deep reinforcement learning, which specifically includes the following steps:

[0063] S1, establish a non-stationary MIMO channel model;

[0064] S2, based on the non-stationary MIMO channel model, modeling the problem of jointly optimizing the MIMO mode and modulation mode as a Markov decision process, and defining the state space, action space, transition probability and reward function therein;

[0065] S3, using the PPO deep reinforcement learning algorithm to solve the Markov decision process to obtain the optimal MIMO mode and modulation method;

[0066] Specifically,

[0067] In step S1, this embodiment considers Figure 2 The MIMO transmission system model shown in the figure considers the case where uniform linear arrays (ULA) are deployed at the transmitter and receiver, where the number of antennas at the transmitter and receiver are N respectively. t and N r ; The channel considered is a time-varying non-stationary channel. The transmission bandwidth of the signal is much smaller than the coherence bandwidth, and the signal duration is much smaller than the correlation time. It is a quasi-static channel within the transmission time of each signal frame, and the channel has non-stationary characteristics between each signal frame; therefore, the received signal of the tth frame can be expressed as follows:

[0068] y(t)=H(t)x(t)+z(t)

[0069] in, represents the received signal vector, y n (t) represents the received signal of the nth antenna, represents the transmitted signal vector, x m (t) represents the transmitted signal of the mth antenna. represents additive Gaussian white noise, which obeys represents the channel model between transceivers, N t Indicates the number of antennas at the transmitting end, N r Indicates the number of antennas at the receiving end.

[0070] In the channel model H(t) between the transceivers, the element h in the mth row and nth column is mn(t) represents the channel between the mth transmitting antenna and the nth receiving antenna; considering the large-scale fading caused by path loss and shadowing and the small-scale fading caused by multipath propagation, h mn The expression of (t) can be written as:

[0071]

[0072] Where PL = 32.5 + 20lg (f c )+20lg(d s )[dB], represents the free space loss, f c Indicates the carrier frequency, d s represents the communication distance, ξ(t) represents the Rice factor, Q represents the total number of non-line-of-sight components, represents the birth and death state of the line-of-sight component, represents the birth and death state of the qth non-line-of-sight component, z q (t)∈{0,1}, represents the birth and death state of the multipath component, which is used to characterize the blocking of the link due to obstacles, where "0" represents the disappearance of the link and "1" represents the existence of the link. represents the sight distance component, represents the qth non-line-of-sight component.

[0073] The transition of multipath link component states can be modeled using a two-state Markov chain. The multipath component two-state Markov chain transition model is as follows: Figure 3 As shown; the persistence and transition of the two states "birth" and "death" are described by the state transition matrix (State Transition Matrix, STM) and the steady state matrix (Steady State Matrix, SSM); the state transition matrix STM and the steady state matrix SSM can be expressed as:

[0074]

[0075] Among them, P ij =Pr(z k =j|z k =i), i,j = {0,1} represents the probability of transitioning from state i to state j. The state transition probability determines the distribution of the birth and death states in the stable state. That is, after T transitions, the birth and death state probability of the link can be calculated by the formula:

[0076] Assuming that the multipath link state does not change within a signal frame transmission time, t can be omitted for convenience of representation; represents the qth non-line-of-sight component, which obeys a complex normal distribution with mean 0 and mean 1; the line-of-sight component It can be expressed as:

[0077]

[0078] Among them, the wavelength λ c =C / f c , C is the speed of light, f c is the carrier frequency, d is the antenna spacing, θ D and Respectively represent the horizontal angle and vertical angle of AOD, θ A and Respectively represent the horizontal angle and vertical angle of AOA. The antenna steering vectors of the transmitting and receiving ends can be expressed as:

[0079]

[0080]

[0081] Therefore, the non-stationary channel model can be expressed as:

[0082]

[0083] in, is the line-of-sight component matrix, is the non-line-of-sight component in the Rice fading channel, each element of which obeys

[0084] Let k(t) represent the MIMO mode of the t-th signal frame. Therefore, for the t-th signal frame, the selectable MIMO modes are as follows: BF mode (k(t) = 1), when there is a strong LoS transmission path between the transmitting and receiving antenna arrays, the signal of the transmitter can directly reach the receiver, and the MIMO channel correlation is strong. The strong correlation of the channel and the principle of electromagnetic interference can be utilized to generate a beam with spatial directivity through weighted processing of the antenna array, so that the energy of the transmitted signal is focused on the receiving antenna, thereby improving the received signal-to-noise ratio and obtaining the array gain; SD mode (k(t) = 2), in a rich scattering and low signal-to-noise ratio environment, the same information is sent and transmitted on different antennas, and after independent channel fading, it reaches the receiving end, thereby counteracting channel fading and improving the reception reliability; SM mode (k(t) = 3), in a rich scattering and high signal-to-noise ratio environment with a high degree of freedom, independent information is transmitted in parallel in the spatial channel through spatial multiplexing technology to improve the data transmission rate.

[0085] The spatial degrees of freedom of the MIMO system is determined by the rank of the non-stationary channel model H. The singular value decomposition of the MIMO channel can be expressed as:

[0086] H=U∑V

[0087] in, and is an orthogonal matrix, where rank{H} is the rank of H and satisfies UU H =U H U=I,VV H =V H V=I, ∑ is a diagonal matrix composed of non-zero singular values, and its expression can be written as: ∑=diag{[λ1,λ2,···λ rank{H} ]}.

[0088] Define a binary variable b m ={0,1} to indicate whether the m-th modulation method meets the bit error rate requirement for reliable transmission. The expression is as follows:

[0089]

[0090] Among them, P e represents the bit error rate, P e,th is the target bit error rate, which is set according to the actual needs of the communication scenario. Therefore, the spectrum efficiency of the tth signal frame is expressed as:

[0091]

[0092] Among them, η m is the spectral efficiency of the mth modulation method.

[0093] Based on the above analysis, the transmitter needs to jointly optimize the MIMO transmission mode and modulation order according to the channel conditions; in a non-stationary communication environment, the transition probability of the channel state is unknown and difficult to predict, which brings great difficulties to the design of the transmission strategy.

[0094] In step S2, this embodiment models the dynamic decision problem as a Markov decision process, which can be mathematically represented as a 4-tuple of state, action, transition probability and reward function: In the tth signal frame, for the channel state The agent performs actions according to the existing strategy Get rewarded And according to the transition probability Get the next state The specific modeling is as follows:

[0095] (1) State space The state of the tth signal frame can be defined as Where (t) and SNR(t) represent the spatial correlation coefficient of the channel model and the estimated signal-to-noise ratio, respectively;

[0096] In order to reduce the feedback information, the spatial correlation between the transmitting and receiving antennas can be approximated by the following formula:

[0097]

[0098]

[0099] Among them, diag m [·] and diag n [·] indicates that the m-th and n-th diagonals of the channel model are vectorized. m [·]} l and {diag n [·]} l represents the lth element in the vector obtained after vectorization. The correlation vector of the MIMO matrix can be expressed as:

[0100]

[0101] In an actual MIMO transmission system, at the receiving end, statistical MIMO channel information can be obtained by estimating the received signal-to-noise ratio and calculating the spatial correlation coefficient, and then transmitted to the transmitting end through a feedback link. When the number of antennas is large or the channel changes rapidly, the overhead of directly feeding back the multi-antenna channel model is too large and too frequent, which will bring about a large overhead. Compared with the transmitted data, the MIMO statistical channel feedback information composed of the estimated signal-to-noise ratio and the correlation vector can be approximately considered to have no propagation error.

[0102] (2) Action Space The action of the tth signal frame can be defined as a t ={k(t),m(t)}, where k(t) and m(t) represent the MIMO transmission mode and modulation mode respectively; the optional MIMO transmission modes are The available modulation modes are: So the action space The size can be expressed as

[0103] (3) Transition probability The transition probability is used to describe the change of state. t Next, perform action a t After that, in s t+1 The probability of is defined as the transition probability, that is In this embodiment, the transition probability is used to describe the change of the non-stationary channel environment.

[0104] (4) Reward Function Reward function r t is the state of the agent in s t When , execute action a t After that, the reward obtained by interacting with the environment can be expressed as The goal of the agent is to maximize the transmission performance of the MIMO transmission system by adjusting the MIMO transmission mode and modulation mode. Among the performances considered, two performance indicators are mainly involved, namely, spectrum efficiency and bit error rate. The goal is to obtain higher spectrum efficiency as much as possible under the premise of ensuring transmission reliability, that is, to select a suitable mode to maximize spectrum efficiency under the condition of meeting the preset bit error rate threshold. Therefore, this embodiment designs the reward function based on the following principles: when the bit error rate is lower than the preset threshold P e,th When P , it can be considered that the transmission system meets the reliability requirements. At this time, the agent will perform the action with the highest spectrum efficiency, and the agent can obtain an additional positive reward r1. e >P e,th , then the reliability transmission condition is not met, the spectrum efficiency is 0, and the agent will receive an additional penalty -r2; the reward function is defined as follows

[0105]

[0106] Among them, r t (s t ,a t ) indicates that the agent is in state s t When , execute action a t After that, the reward obtained by feedback after interacting with the environment, η(t) represents the spectral efficiency of the t-th signal frame, r1 represents the positive reward, r2 represents the penalty, P e represents the bit error rate, P e,th Indicates the preset bit error rate threshold.

[0107] In order to avoid the agent from making wrong actions, let r2 = 5r1; the optimization goal is to find the optimal transmission strategy π * (a|s) to maximize the expected discounted reward function during the entire communication period, so the optimization problem can be expressed as:

[0108]

[0109] Where T is the entire communication transmission time, and 0<γ<1 is the discount factor used to describe the importance of current and future rewards.

[0110] In step S3, the PPO algorithm is implemented based on the actor-critic (AC) framework, which includes a critic neural network (with parameters θ c ), an old actor network (with parameters θ a,old ), a new actor network (with parameters θ a,new); optimize the agent's strategy through policy gradient, where the strategy can be fitted by neural network, and the neural network parameters can be adjusted to obtain the optimal strategy, so as to obtain the maximum reward; when the reward is positive, the probability of the corresponding action is increased; conversely, if the reward function is negative, the corresponding action probability is reduced. Therefore, by minimizing the loss function, optimizing the neural network parameters, and increasing the probability of selecting actions that can obtain larger rewards, a strategy network that maximizes rewards is obtained.

[0111] The actor network is used by the agent to select actions. It consists of an input layer, two hidden layers and an output layer. Its network parameters are denoted by θ a ; Its input and output are the MIMO channel state s t and the vector π(a,s t θ a ), the normalized action probability vector is The sum of its elements is 1; the critic network is used to evaluate the performance of the current strategy. It contains an input layer, whose input and output are state s t and the estimated value V of the state-action function π (s t θ c ), its expression can be written as

[0112]

[0113] Since it is difficult to determine the learning rate and step size for the policy gradient algorithm, the large difference between the new and old strategies will lead to poor learning results. In order to avoid the optimal strategy being replaced by the suboptimal strategy, PPO introduces an advantage function in the AC framework to measure the quality of the action, and uses generalized advantage estimation (GAE) to reduce the influence of the bias in the advantage function. Its expression can be written as

[0114]

[0115]

[0116] Among them, λ∈[0,1] is the GAE factor, which is used to control the size of bias and variance. , it means that the current action is better, otherwise, it is believed that the current action cannot obtain a good reward function.

[0117] When updating, the old actor network is based on the strategy π(a|s;θ old ) interacts with the environment, obtains data, and passes it to the new actor network for strategy π(a|s;θ new) update; in the update phase, the agent extracts historical data of mini-batch size from the cache to update the network parameters; in the policy descent algorithm, there is a problem that the learning rate and step size are difficult to determine; in order to prevent excessive updates or slow updates, the probability ratio of the new and old strategies is introduced to control the update amplitude of the new strategy, and its expression can be written as:

[0118]

[0119] Therefore, the network parameter gradient update expression in the PPO algorithm can be written as:

[0120]

[0121] In order to satisfy the trust region constraint, the objective function is pruned to reduce the difference between the new and old strategies, which can be further expressed as:

[0122]

[0123] Where, clip(·) represents the clip function, ε clip is a hyperparameter, clip(ρ t (θ a ),1-ε clip ,1+ε clip ) represents the t (θ a ) is limited to (1-ε clip ,1+ε clip ).

[0124] In order to improve the generalization ability of the proposed model in different channel scenarios and increase the exploration ability of the agent, the policy entropy H(π(·|s t )), while optimizing the objective function, the entropy of the strategy is increased, and the exploration ability of the algorithm is improved. Its expression can be written as:

[0125]

[0126] The loss functions of the actor network and the critic network can be expressed as:

[0127]

[0128] L critic (θ c )=E π [(V tar (s t θ c )-V π (s t θ c ))2 ],

[0129] Where V tar (s t θ c ) is the target value function of the critic network, and its expression can be expressed as: V tar (s t θ c )=r t (s t ,a t )+γV π (s t+1 θ c ).

[0130] Through stochastic gradient descent, the parameters of the actor and critic networks are optimized as follows:

[0131]

[0132]

[0133] Among them, l a and l c Represent the learning rates of actor and critic networks respectively, and L actor and L critic The specific implementation of the i-th tuple in the mini-batch. m is the size of the mini-batch.

[0134] like Figure 4 The following is a schematic diagram of the PPO network. Figure 1 As shown, the specific steps of the PPO-based adaptive MIMO mode and modulation method selection method are as follows:

[0135] Step 1: Initialize the neural network parameters θ a,new ,θ a,old ,θ c , the experience playback unit capacity is N E , the number of cycles e = 1, the maximum number of cycles E max ;

[0136] Step 2: Let θ a,old ←θ a,new ;

[0137] Step 3: Initialize state s0;

[0138] Step 4: Iteration number t = 1, maximum iteration number T max ;

[0139] Step 5: The PPO agent observes the environment state s t , input actor-critic network;

[0140] Step 6: The critic network outputs the value function V π (s t θ c ), the actor network outputs the action selection strategy π(a|s t θ a,old );

[0141] Step 7: The PPO agent selects action a according to the action selection strategy t , and transfer to the next state s t+1 ;

[0142] Step 8: Evaluate the bit error rate performance and spectrum efficiency and calculate the reward function r t ;

[0143] Step 9: Substitute the tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience playback unit;

[0144] Step 10: If Mod(t+1,B m )==0, then sample the historical data of Mini-batch from the experience playback unit to calculate the advantage function, and update the network parameters θ by gradient descent a,new , and let θ a,old ←θ a,new Otherwise, the network parameters will not be updated;

[0145] Step 11: t = t + 1, if t ≤ T max , then return to step 5, otherwise end;

[0146] Step 12: e=e+1, if e≤E max , then return to step 3, otherwise end.

[0147] In order to evaluate the performance of the PPO-Clip scheme proposed in this embodiment, it is compared with four other conventional baseline methods:

[0148] BER-greedy solution: Give priority to meeting the bit error rate requirements, and use the SD mode or BF mode of low-order BPSK under RSE or LDE conditions to ensure the bit error rate requirements.

[0149] SE-Greedy solution: pursues spectrum efficiency and adopts high-order SM mode to pursue spectrum efficiency.

[0150] Fixed BF solution: The BF mode is fixedly adopted to directly improve the received signal-to-noise ratio by using the beam, and the modulation method adopts QPSK. However, under the RSE condition, the bit error rate requirement cannot be met.

[0151] Random scheme: Regardless of the channel characteristics, a transmission mode and modulation method are randomly selected for the current channel.

[0152] The simulation parameters are set to the number of days the antenna transmits and receives data as N. t =N r =4, the transmission power is 5W, P e,th =10 -3 , the signal-to-noise ratio is randomly distributed in [-20dB, 30dB]; reference Figure 5 It can be seen that with the increase of the number of iterations, the average reward of the proposed algorithm continues to increase and converges at about 10,000 times; the PPO-Clip algorithm effectively avoids the performance collapse problem and improves the training stability by clipping the reward function; in addition, compared with the other four benchmark schemes, the proposed algorithm can obtain a higher average reward; this is because after training, the proposed scheme can select an optimal transmission strategy based on the current channel characteristics, adaptively adjust the MIMO transmission mode and modulation mode, and improve the spectrum efficiency while ensuring the bit error rate performance, thereby obtaining the maximum reward function; other schemes cannot adapt to the non-stationary channel transmission environment, resulting in lower rewards. The BER-greedy scheme prioritizes the bit error rate constraint, so the spectrum efficiency of this scheme is low; the SE-greedy scheme can meet the bit error rate requirements when the signal-to-noise ratio is high, and the spectrum efficiency is also the highest; but under the condition of low signal-to-noise ratio, it cannot meet the bit error rate requirements; the fixed BF scheme can only obtain gains under the channel conditions dominated by the LoS component, and its performance is poor in the rich scattering environment; the random scheme does not consider the characteristics of the channel and cannot guarantee the bit error rate and spectrum efficiency performance.

[0153] Figure 6 Given different transmission powers P t Impact on the proposed algorithm; It can be found that the proposed algorithm can converge under different transmission power conditions; and the greater the power, the greater the reward of the proposed algorithm; This is because, the greater the power, in the BF mode, it can provide a higher power gain for the beam, in the SD and SM modes, each subchannel can obtain a better channel gain, thereby obtaining a higher signal-to-noise ratio, adopting a higher-order modulation method, and obtaining a larger reward function.

[0154] In summary, this embodiment proposes a MIMO and modulation joint optimization solution based on proximal policy optimization (PPO) to solve the above-mentioned MDP problem; the transmitter, as an intelligent agent, interacts with the environment, inputs states into the neural network model, outputs actions, and obtains the maximum reward; through the feedback reward obtained, the complex MIMO channel environment is feature extracted and cognitively understood, the neural network parameters are optimized, the optimal action adapted to the current MIMO channel environment is found, and it is applied to guide the transmission selection of the transmitter.

[0155] refer to Figure 4 As shown in Figure 1, first, the MIMO channel correlation and channel average gain are input into the deep neural network as states; the deep neural network extracts and learns the relevant feature relationships of the non-stationary MIMO channel, and selects the appropriate MIMO transmission mode and modulation method as the action of the agent; next, the transmitter interacts with the environment according to the selected action and obtains reward feedback; in each time slot, the agent observes the channel state information s t ∈S, executes the action a selected by the actor network t ∈A, get reward r t , and enter the next state; the agent will tuple {s t ,a t ,r t ,s t+1}Stored in the experience playback unit.

[0156] Embodiment 2:

[0157] This embodiment provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the method for selecting a MIMO mode and modulation method based on deep reinforcement learning as described in any one of the first embodiments are implemented.

[0158] Embodiment three:

[0159] This embodiment provides a computer device / equipment / system, including:

[0160] Memory, for storing computer programs / instructions;

[0161] A processor is used to execute the computer program / instructions to implement the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning as described in any one of the first embodiments.

[0162] Embodiment 4:

[0163] This embodiment provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning as described in any one of the embodiments.

[0164] In summary, the MIMO mode and modulation mode selection method based on deep reinforcement learning provided by the present invention has the following beneficial effects:

[0165] 1. Jointly optimize the MIMO transmission mode and modulation method, taking into account the transmission conditions of non-stationary channels and maximizing the spectrum efficiency while ensuring transmission reliability;

[0166] 2. The algorithm takes into account the selection of three transmission modes: BF, SD, and SM, and optimizes the modulation mode to ensure the performance of MIMO technology in non-stationary transmission environments. Based on MIMO transmission characteristics and optimization goals, the corresponding state, action, and reward functions are designed respectively;

[0167] 3. It can effectively solve the problems of step size and learning rate, ensure the convergence of the algorithm, use neural network to extract, analyze and reason the features of complex MIMO channel environment, match the corresponding multi-dimensional communication parameters according to the obtained channel features, and design a joint transmission strategy suitable for the current MIMO channel environment;

[0168] 4. Simulation results show that compared with the benchmark solution, this method can significantly improve the spectrum efficiency while meeting the reliability requirements.

[0169] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A MIMO mode and modulation selection method based on deep reinforcement learning, characterized in that: include: Establish a non-stationary MIMO channel model; According to the non-stationary MIMO channel model, the problem of jointly optimizing the MIMO mode and modulation mode is modeled as a Markov decision process, and the state space, action space, transition probability and reward function are defined therein; The PPO deep reinforcement learning algorithm is used to solve the Markov decision process to obtain the optimal MIMO mode and modulation method; The expression of the joint optimization of MIMO mode and modulation method after Markov decision process modeling is as follows: ； in, Represents the entire communication transmission time, , represents the discount factor, which is used to describe the importance of current and future rewards, Indicates that the agent is in state When Finally, the reward obtained after interacting with the environment; The state space In The status of a signal frame is ,in, represents the spatial correlation coefficient of the channel model, represents the estimated signal-to-noise ratio of the channel model; The action space In The action of a signal frame is ,in, , represents the MIMO transmission mode, , represents the modulation mode, action space The size is ; The transition probability Used to describe the changes in non-stationary channel environment. Next, execute the action After that, in The transition probability is .

2. The method for selecting MIMO mode and modulation scheme based on deep reinforcement learning according to claim 1, characterized in that: The expression of the non-stationary MIMO channel model H is as follows: ； in, represents the free space loss, represents the Rice factor, and Both represent additive Gaussian white noise, which obeys , represents the noise variance The dimension is The unit diagonal matrix of represents the line-of-sight component matrix, represents the horizontal angle of AOD, represents the vertical angle of AOD, and represents the antenna steering vector at the transmitting and receiving ends, represents the horizontal angle of AOA, Indicates the vertical angle of AOA, represents the total number of non-line-of-sight components, represents the non-line-of-sight component in the Rice fading channel, each element of which obeys .

3. The method for selecting MIMO mode and modulation scheme based on deep reinforcement learning according to claim 2, characterized in that: The reward function The expression is as follows: ； in, Indicates that the agent is in state When After that, the reward obtained by interacting with the environment is Indicates The spectral efficiency of a signal frame, Indicates positive rewards, Indicates punishment, represents the bit error rate, Indicates the preset bit error rate threshold.

4. The method for selecting MIMO mode and modulation scheme based on deep reinforcement learning according to claim 3, characterized in that: The spatial correlation coefficient of the channel model The expression is as follows: ； in, Indicates the number of antennas at the transmitting end, Indicates the number of antennas at the receiving end, , represents the spatial correlation of the transmitting antenna, , represents the spatial correlation of the receiver, and Represents the channel model The diagonal and The diagonals are vectorized. and Represents the first elements, H represents the non-stationary channel model, represents the conjugate transpose of the matrix H.

5. The method for selecting MIMO mode and modulation scheme based on deep reinforcement learning according to claim 1, characterized in that: The use of the PPO deep reinforcement learning algorithm to solve the Markov decision process includes the following steps: Initialize neural network parameters , set the experience playback unit capacity to ,Number of cycles , maximum number of cycles ; make ; Initialization state ; Let the number of iterations , set the maximum number of iterations ; Observe the environmental status , and the environmental state Input the actor-critic network so that the critic network outputs the value function , the actor network outputs the action selection strategy ; Select a strategy based on the action , perform the action , and transfer to the next state ; Evaluate bit error rate performance and spectral efficiency, and calculate reward function ; The tuple Stored to the experience playback unit, and make the following judgment: , then sample the historical data of Mini-batch from the experience playback unit to calculate the advantage function, and update the network parameters by gradient descent ,make ; Otherwise, the network parameters are not updated; make ,like , then return the observation environment status Step, execute the next iteration, otherwise end; make ,like , then return to the initialization state If the next iteration is completed, the next iteration is executed; otherwise, the process ends.

6. The method for selecting MIMO mode and modulation scheme based on deep reinforcement learning according to claim 5, characterized in that: The advantage function is used to measure the quality of an action, and its expression is as follows: ； in, represents the ubiquitous advantage estimate, , represents the discount factor, , represents the GA factor, which is used to control the size of bias and variance, , Indicates that the agent is in state When After that, the reward obtained by interacting with the environment is Indicates that the agent is in state The value function when Indicates that the agent is in state The value function when when , it means that the current action is better, otherwise, it is considered that the current action cannot obtain a good reward function.

7. The method for selecting MIMO mode and modulation scheme based on deep reinforcement learning according to claim 6, characterized in that: The network parameters The expression for gradient descent update is as follows: ； in, represents the action selection strategy, represents the magnitude of the policy update, represents the advantage function, express function, represents the hyperparameter, Indicates that The value is limited to .

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning described in any one of claims 1-7 are implemented.

9. A computer device / equipment / system, characterized in that: include: Memory, for storing computer programs / instructions; A processor, configured to execute the computer program / instructions to implement the steps of the MIMO mode and modulation selection method based on deep reinforcement learning as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the MIMO mode and modulation method selection method based on deep reinforcement learning described in any one of claims 1-7 are implemented.

Images