An intelligent decision-making method for physical layer parameters

Abstract

The present invention relates to an intelligent decision method for physical layer parameters, which belongs to the field of mobile communication technology and includes the following steps: S1: inputting the parameters of the RLC layer, MAC layer, and PHY layer as cross-layer information, using an intelligent scheduler to dynamically decide and output the modulation order of the physical layer according to different channel conditions while satisfying the upper layer constraints; S2: modeling the optimization problem as a Markov decision process, using DRL to design an intelligent agent, defining the state space, action space, and reward function; S3: using DQN to verify the intelligent decision-making scheme for flexible configuration of physical layer parameters in a complex network environment. The present invention makes the scheduling of the MAC layer more intelligent, and for different network environment status information, the method proposed by the present invention can effectively solve the problem of intelligent decision-making for flexible configuration of physical layer parameters in a complex network environment.

Description

Technical Field

[0001] The invention belongs to the technical field of mobile communications and relates to an intelligent decision method for physical layer parameters. Background Art

[0002] With the explosive growth of mobile terminal devices and network data traffic, various new businesses continue to emerge, putting higher requirements on communications and information. The world's first 6G white paper in 2019 proposed that future communications will be deeply integrated with artificial intelligence to achieve intelligent resource allocation, and the application trend of communication technology will gradually develop in the direction of intelligence and adaptability.

[0003] Traditional communication protocols are built in layers, so the design of network protocols can be simple and standardized. However, with the improvement of the intelligence of wireless communication systems, the intelligence of the end-to-end communication system protocol layer has also been improved accordingly. The realization of many functions requires the simultaneous mobilization of parameters of multiple layers. In a wireless network environment, overall wireless resource management can be achieved through cross-layer optimization between the link layer, network layer, transport layer and physical layer, thereby improving network performance. The physical layer is the lowest layer for device interconnection. Key parameters include bandwidth, cyclic prefix (CP), subcarrier spacing, transmission time interval (TTI), modulation and coding scheme (MCS), transmission power, etc., which should be dynamically established and adjusted according to application requirements or network layer conditions under the premise of supporting high traffic mode. It plays a very important role in the fields of IoT security, vehicle communication, optical communication, autonomous driving, smart medical care, etc. The configuration of physical layer parameters is often determined by information from multiple layers including link layer, network layer and transport layer. This is actually a problem in cross-layer optimization that uses models to dynamically meet target optimization issues, such as maximizing throughput, minimizing latency, or minimizing power output. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a medium access control intelligent decision (MID) method for physical layer parameters, which transforms the decision-making problem into a deep reinforcement learning (DRL) process, and intelligently decides the physical layer modulation parameters through real-time interaction of environmental variables at each layer of the current network, so as to optimize the transmission speed of the communication system while satisfying the constraints of the upper layer, so that the intelligent agent of the MAC layer can pay attention to the appropriate physical layer modulation parameters in the current network environment in an effective state.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for intelligent decision-making of physical layer parameters comprises the following steps:

[0007] S1: Input the parameters of the RLC layer, MAC layer, and PHY layer as cross-layer information, and use the intelligent scheduler in the MAC to dynamically decide and output the modulation order of the physical layer according to different channel conditions while satisfying the constraints of the upper layer;

[0008] S2: The MID model models the optimization problem as a Markov decision process, uses the DRL algorithm to design the agent, and defines the state space, action space, and reward function;

[0009] S3: Using the designed intelligent agent, Deep Q-Network (DQN) is used to verify the intelligent decision-making scheme for flexible configuration of physical layer parameters in complex network environments.

[0010] Furthermore, in step S1, the buffer status of RLC, the HOL delay of MAC and the channel quality of PHY are used as cross-layer input information; the cross-layer information is transmitted through logical channels and transmission channels, and is used as a dynamically changing network environment to interact with the intelligent agent of MAC in real time.

[0011] Furthermore, the upper layer constraints in step S1 include whether there is data in the Buffer state of the RLC layer and whether the HOL delay time of the MAC layer exceeds the delay threshold; the intelligent scheduler of the MAC layer performs resource scheduling only when there is data in the buffer, and it is invalid when there is no data; the same HOL delay threshold is set for each user request, and the data packets exceeding the HOL delay threshold are discarded from the Buffer; the different channel conditions are represented by the signal-to-noise ratio, and the channel quality is divided into three conditions: low, medium and high.

[0012] Furthermore, the optimization goal of step S1 is to configure the appropriate physical layer modulation order for the current channel environment by defining the maximum number of transmitted symbols under the constraints of buffer status information and HOL delay. The delay constraint only works when there is data. The function is defined as To regulate the delay constraint, the function variable is d t -d th , when the remaining time for the jth requested data packet to be scheduled in the Buffer is less than the HOL delay threshold, it is considered to meet the delay constraint, otherwise it is not satisfied. The mathematical model of the system maximizing the number of symbols is expressed as:

[0013]

[0014] Where R m,j (t) represents the number of symbols transmitted by the system, f(d t -d th ) represents the delay constraint function, d t represents the delay time of the data packet in the buffer at time t, d t -d th represents the difference between the delay time of the data packet at time t and the delay threshold. st represents the constraint condition, and the set T = {0, 1, ..., T-1} represents the set of all decision times of the current UE. represents the total channel capacity of the system, It represents the minimum channel capacity of the system, C1 is the minimum delay limit, that is, to ensure that resources are scheduled while meeting the delay constraint, and C2 represents the minimum service requirement of the network, that is, to ensure that the total channel capacity of the current UE cannot be lower than the throughput threshold.

[0015] Further, in step S2, the MID model designs a Markov decision process MDP model so that the intelligent agent of the MAC layer can decide the physical layer modulation order suitable for the current network environment under different states. The MDP model is designed as follows:

[0016] First, the state is abstractly represented as the environment, that is, the basis for the agent to make decision modulation orders. The state of MDP at time t is defined as:

[0017] S(t)={B(t),D(t),H(t)}

[0018] After observing the environment, the agent obtains the corresponding state S(t) and determines the action A(t) to be performed in this state. The result of the decision is a set of modulation order vectors that satisfy the constraint condition C1. The action space is defined as

[0019] A(t)=(m(t))

[0020] where m(t)={m1(t),m2(t),...,m j (t)|m j (t)∈[4,16,64]},m j (t) represents the modulation order decided by the agent when the UE makes the jth request;

[0021] Reward is the immediate feedback after the agent takes action in the MDP. The delay constraint function is used as a delay multiplication factor to reward the situation where the constraint is met. If the delay constraint is met, a positive reward is obtained. If it is not satisfied, a negative penalty is obtained. By defining an adjustment factor δ to amplify the reward when the optimal action is taken, when δ is less than 1, the reward is amplified by β times. By amplifying the reward by β times, the agent pays more attention to the optimal action. β is defined as follows:

[0022]

[0023] The reward function R(t) is defined according to the defined delay constraints and the state information of the cross-layer parameters. When there is no data to be scheduled in the Buffer, the reward is 0; when there is data to be scheduled in the Buffer, the goal of the system is to accumulate and maximize the positive reward; for low signal-to-noise ratio, the optimal action is low-order modulation, and the adjustment factor is set to be less than 1 when the agent selects the low-order modulation; similarly, the modulation action under medium and high signal-to-noise ratio conditions is considered as the appropriate action for medium and high-order modulation, respectively, and the reward function R(t) is:

[0024]

[0025] Where B represents the bandwidth of the system.

[0026] Further, in step S3, the DQN uses a nonlinear neural network to approximate the state-action value function, extracts features from high-dimensional data, and completes the mapping from state to state-action value. The experience replay pool is responsible for storing the 4-tuple samples obtained through interaction, and works together with the target network and the adaptive moment estimation (Adam) optimizer to train the Q network. The Q value function update formula of DQN is as follows:

[0027]

[0028] Q * (s,a) is the optimal Q-value function, obtained by recursion. (s(t),a(t),r(t),s(t+1),a(t+1)) means that at the current time t, the system is in state s(t), the immediate reward obtained after executing action a(t), and enters state s(t+1) at the next time with a certain probability to execute action a(t+1). The Q-value function update rule is as follows:

[0029]

[0030] Among them, α represents the learning rate, γ∈[0,1] is the discount factor, and the discount factor reflects the network's expectation of obtaining immediate rewards. The experience replay pool is used to store the obtained 4-tuple samples, and small batches of sample data are randomly selected for training.

[0031] Furthermore, during the training process, the DQN network trains and updates parameters by minimizing the loss function and gradually approaching the optimal Q function. The loss function is defined by the difference between the current Q value and the target Q value. The correct direction of the next training step is determined by evaluating this difference. The loss function calculation formula is as follows:

[0032]

[0033] Where r(t) is the reward at the current moment; w- and w are the parameters of the target network and the evaluation network respectively;

[0034] Backward training update of Q network by gradient descent method:

[0035]

[0036] Furthermore, the DQN model is trained. After the training is completed, the optimal weight parameter w of the DQN network is obtained, and the optimal decision modulation strategy is achieved by maximizing the number of symbols in the system.

[0037] The beneficial effects of the present invention are as follows: the present invention adopts a DQN network and proposes a MID model. By designing a DRL Markov decision model, the scheduling of the MAC layer is made more intelligent. For different network environment status information, the method proposed by the present invention can effectively solve the intelligent decision-making problem of flexibly configuring physical layer parameters in a complex network environment.

[0038] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0040] Figure 1 A framework for intelligent decision making;

[0041] Figure 2 It is an intelligent decision-making framework based on DRL;

[0042] Figure 3 It is the DQN algorithm model;

[0043] Figure 4 Performance under different learning rates;

[0044] Figure 5 is the performance under different discount rates;

[0045] Figure 6 is the loss graph;

[0046] Figure 7 For different adjustment multiples of performance. DETAILED DESCRIPTION

[0047] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0048] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0049] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0050] See also Figure 1 to Figure 7The present invention provides a Medium Access Control Intelligent Decision (MID) model. The model transforms the decision problem into a deep reinforcement learning (DRL) process, and through real-time interaction of the current network environment variables of each layer, intelligently decides the physical layer modulation parameters, and optimizes the transmission speed of the communication system under the constraints of the upper layer. MID models the optimization problem as a Markov decision process (MDP) and designs a DRL agent. By designing the MDP model in reinforcement learning, the state information of the buffer of the wireless link control layer, the HOL delay time information of the media access control layer and the channel quality information of the physical layer are defined as the state space, and the physical layer modulation order is used as the action space. MID designs a corresponding reward function according to the change in the number of transmission symbols under the delay constraint condition in unit time, and designs an adjustment factor in the reward function, so that the MAC layer agent can pay attention to the appropriate physical layer modulation parameters in the current network environment in an effective state.

[0051] The Medium Access Control Intelligent Decision (MID) model framework used in the present invention is as follows: Figure 1 As shown in the figure, the base station (BS) makes scheduling decisions based on the service requests of the user (UE). After the UE sends multiple service requests to the base station BS, the upper layer encapsulates them into messages using the TCP / UDP protocol and defines them as IP data packets through the IP protocol. The service request data packets arrive at the RLC layer buffer, and its queue model follows the first-in-first-out principle. The MAC layer scheduler in 5G NR needs to comprehensively evaluate multiple layers of parameter information, such as Figure 1The parameters of the RLC layer include buffer size, buffer status, ARQ, etc., the time slot, frame size, HOL delay, etc. of the MAC layer, and the channel conditions, power, modulation and coding, etc. of the PHY layer. From these parameters, the MAC scheduler analyzes the user's category, buffer status, the amount of data to be scheduled by the user, the QoS content of the service bearer, the channel quality fed back by the user, the transmission power, etc. It is necessary to consider not only the service queue model, but also the physical layer model, that is, the channel conditions of the available radio resources experienced by the user. The user feeds back the channel quality information to the base station to determine the modulation scheme. The scheduler of the MAC layer can use the characteristics of the interaction between the agent and the environment in reinforcement learning to complete the intelligent design. The cross-layer parameter information is transmitted through the logical channel and the transmission channel. The cross-layer parameter information is used as a dynamically changing network environment and interacts with the MAC agent in real time.

[0052] The present invention first considers the buffer state parameters of the RLC layer to represent the dynamic change process of the data packet state in the queue. The MAC layer scheduler intelligently allocates resources based on whether there is data information in the buffer. When the UE makes a request, each UE's request is sent to the BS as a tuple, which includes the service type it requires, some predefined parameters, and its quantized channel estimate. The buffer request is indexed as j∈{1,2,...,n}, and the buffer state requested at time t can be expressed as B(t)={q j (t)|j∈{1,2,...,n}}, the initially requested Buffer is empty, that is, B(0)={q j (0) = 0|j∈{1,2,...,n}}, if the buffer is full, the reception of new data packets will be rejected.

[0053] Then consider the HOL delay time parameter information of the MAC layer. When the MAC layer scheduler performs scheduling, it determines whether to perform scheduling based on the queue status of the RLC layer buffer. When the buffer has data, it will schedule. Otherwise, it will wait for the next request from the user to arrive before scheduling. The HOL delay of the MAC layer is represented by D(t) = {d1(t), d2(t), ..., d j (t),...d n(t)}, it must be lower than the specified delay threshold. Packets that violate the preset HOL delay threshold will be discarded from the Buffer. Due to the diversity of channel quality and data rates of different applications, some packet flows may experience higher HOL delay or less HOL delay. The HOL delay threshold can be set for application scenarios with different delay requirements. In this paper, for fairness considerations, regardless of the priority of the request, the HOL delay threshold of each user request is set to the same. Under the constraints based on the Buffer state parameters and HOL delay, when allocating resources to the current user, it is necessary to combine the user's instantaneous signal-to-noise ratio value and the channel condition parameters of the physical layer.

[0054] Finally, consider the channel condition parameter information of the physical layer. Generally speaking, the channel between the BS and the UE is a time-frequency varying channel. The channel vector corresponding to the jth request in the buffer at time t is represented by . Each new request is sent by the UE, and the UE may have a new channel condition at different times. The signal received by the UE for the jth request is represented by

[0055] y j (t) = h j (t)x j (t)+z j (t)

[0056] x j (t) is the transmission signal, z j (t) is additive white Gaussian noise, h j (t) is the channel coefficient, assuming that the user j-th request can always get its channel vector h j The instantaneous SNR observed by the user at the jth request at t can be expressed as

[0057]

[0058] Consider the signal-to-noise ratio as a parameter for evaluating channel quality. The user's channel capacity can be expressed by the Shannon formula:

[0059] C = Blog2(1+SNR)

[0060] When the MAC layer scheduler allocates the physical layer modulation order for the jth user request, the number of symbols transmitted by the system can be expressed as

[0061]

[0062] The present invention configures a suitable physical layer modulation order for the current channel environment by defining the maximum number of transmitted symbols under the constraints of buffer status information and HOL delay.

[0063] From the above analysis, we can see that the Buffer status is related to the number of data packets scheduled at the previous moment and the remaining time to be scheduled. Therefore, the change process of the Buffer has Markov properties and can be modeled as a Markov decision process. The system status, action, and reward are set as follows.

[0064] The state can be abstractly represented as the environment, which is the basis for the agent to make decisions on the modulation order. For a specific moment, the Buffer status of the RLC layer, the queue HOL delay threshold of the MAC layer, and the channel conditions of the PHY layer are used as the environment in the DRL. The cross-layer parameter information is input into the Deep Neural Network (DNN), and the optimal action is output by learning the optimal strategy. Therefore, the state of the MDP at time t can be defined as S(t) = {B(t), D(t), H(t)}.

[0065] After observing the environment, the agent obtains the corresponding state and determines the action to be performed in that state. Therefore, the action space can be defined as A(t) = (m(t)), where m j (t) represents the modulation order decided by the agent when the UE makes the jth request.

[0066] Reward is the immediate feedback after the agent in the MDP takes an action. The reward the agent receives after each action is different. There are two types of rewards: one is a positive reward, that is, the current reward is greater than the reward of the previous step; the other is a negative penalty, that is, the current reward is negative. The core of the reward function designed in this paper is to amplify the current positive reward and reduce the current negative reward. First, the delay constraint function defined above is used as a delay multiplication factor to reward the situation where the constraint is met. If the delay constraint is met, a positive reward is obtained. If it is not met, a negative penalty is obtained. Then determine the problem of amplifying the current reward. As analyzed above, when the agent chooses a suitable action in the current environment, the current reward is amplified. If the reward is significantly amplified in this case, then this situation will lead to a lower degree of exploration of the agent, which is not conducive to the convergence of the network. If the amplification is insufficient, the role of the reward and punishment function cannot be fully exerted. Therefore, an adjustment factor is defined to amplify the reward when the optimal action is taken. When δ is less than 1, the reward is amplified by β times. By amplifying the reward by β times, the agent pays more attention to the optimal action. Definition The reward function R(t) is defined according to the defined delay constraints and the state information of the cross-layer parameters. When there is no data to be scheduled in the buffer, the reward is 0. When there is data to be scheduled in the buffer, the goal of the system is to accumulate and maximize the positive reward. This paper uses low, medium, and high signal-to-noise ratios to represent three types of channel quality, and represents them separately. For low signal-to-noise ratios, that is, corresponding to poor channel conditions, the optimal action should be low-order modulation, so the adjustment factor is set to less than 1 when the agent selects the low-order modulation. Similarly, the modulation actions under medium and high signal-to-noise ratio states should be regarded as medium-order and high-order modulations, respectively. The relationship between the adjustment factor and the action is shown in Table 1.

[0067] Table 1

[0068]

[0069] The purpose of reinforcement learning is to obtain the best benefits through learning strategies. As a value function iterative reinforcement learning method, the Q-learning algorithm records the value of state-action pairs through the Q-value table and updates the Q-table based on the Bellman equation. Q-learning has obvious limitations. When the state space or action space is particularly large, it is difficult for the Q-learning algorithm to establish and maintain a huge Q-value table, so it is difficult to record all states and actions. In order to solve the dimensionality curse, Mnih

[20] combined the Q-learning algorithm with deep learning and expanded it into a deep Q-network (DQN). DQN is one of the most popular algorithms. The algorithm model of DQN based on the Pytorch architecture is as follows: Figure 3 As shown. DQN uses nonlinear neural networks to approximate the state-action value function, which can extract features from high-dimensional data, and DQN has been widely used in intelligent sequential decision-making problems. Therefore, the problem of flexibly configuring physical layer parameters is transformed into a deep Q learning process, which maximizes the transmission speed of the system by observing the state of each layer of the current network environment and selecting appropriate modulation allocation decisions under the delay constraint. After the agent observes the environment and obtains the state, it immediately obtains the decision action based on the current state through the neural network and outputs the reward. The Q network completes the mapping from state to state-action value. The experience replay pool is responsible for storing the 4-tuple samples obtained by the interaction, and works together with the target network and the adaptive moment estimation (Adam) optimizer to train the Q network and improve the fitting characteristics of the neural network.

[0070] The instant reward requested by the user is the instant reward of the system at the current moment. In addition to the instant reward, the future cumulative reward must also be considered. Under the premise of satisfying the restriction constraints, in order to maximize the number of system symbols, appropriate physical layer parameters must be selected to maximize the cumulative reward value.

[0071]

[0072] Q * (s,a) is the optimal Q-value function, which is usually obtained recursively. (s(t),a(t),r(t),s(t+1),a(t+1)) represents the instant reward r(t) obtained after the system is in state s(t) at the current time t, and enters the state at the next time to perform action a(t+1) with a certain probability. The Q-value function update rule is as follows:

[0073]

[0074] Among them, α represents the learning rate, γ∈[0,1] is the discount factor, and the discount factor reflects the network's expectation of obtaining immediate rewards. In order to break the correlation between data and alleviate the instability of the Q-value function in nonlinear networks, the experience replay pool is used to store the acquired 4-tuple samples, and small batches of sample data are randomly sampled for training. The correlation of the data is destroyed due to its storage and sampling methods, making the training more stable.

[0075] During the training process, the DQN network mainly trains and updates parameters by minimizing the loss function and gradually approaching the optimal Q function. The loss function is defined based on the difference between the current Q value and the target Q value. The correct direction of the next training step is determined by evaluating this difference. Therefore, the minimum value of the loss function means that the algorithm has reached the optimal value in a certain sense

[22] . In this paper, the loss function is defined as

[0076]

[0077] r(t) is the reward at the current moment; γ is the discount factor; w- and w are the parameters of the target network and the valuation network respectively. The defined loss function can judge the quality of the algorithm. In order to optimize and approximate the action value function, it is necessary to perform reverse training and update the Q network through the gradient descent method.

[0078]

[0079] By using the algorithm in Table 1 to train the DQN model, the optimal weight parameter w of the DQN network can be obtained after training. The optimal decision modulation strategy is achieved by maximizing the number of symbols in the system. The time complexity of the algorithm is O(m×(n1×n2+n2×n3)), where m is the input of the neural network, n is the i Represents the number of neurons in the i-th layer of the neural network.

[0080] In order to illustrate the feasibility and superiority of the present invention, a simulation experiment will be conducted below to determine the system simulation parameters and the parameters in the DQN algorithm according to the 5G NR example. The signal-to-noise ratio range in the 5G NR example is 0-30dB, 0-10dB is defined as a low signal-to-noise ratio, 10-20dB is defined as a medium signal-to-noise ratio, and 20-30dB is defined as a high signal-to-noise ratio. During simulation, signal-to-noise ratio sample values ​​are randomly generated in each range. The simulation data set is generated by Python3.6, and the DQN framework used is pytorch 1.10 version. The bandwidth used in this experiment is 10Mhz, the HOL delay threshold is 60ms, and the adjustment factors are 1 / 4, 1 / 8, and 1 / 12. The training cycle in the DQN used is 4000, the experience pool capacity is 5000, the discount factor is 0.99, the learning rate is 0.0001, the batch training number is 64, and the optimizer algorithm is Adam.

[0081] from Figure 4 It can be seen that the adjustment of the learning rate affects the convergence speed and stability of the optimization target. When other parameters remain unchanged, the simulation is performed for the cases where the learning rate is 0.01, 0.001 and 0.0001. After 4000 iterations, the trend of the optimization target value is as follows: Figure 4 When the value is 0.01 and 0.001, the convergence trend appears quickly due to the excessive learning rate, but the optimal strategy cannot be learned, resulting in subsequent oscillations; when the value is 0.0001, the optimization target rises steadily and has good convergence.

[0082] Figure 5 It can be seen that the impact of different discount factor values ​​is an optional empirical value. For most applications, increasing the discount factor is more beneficial to DQN. When other parameters remain unchanged, the discount rate is simulated as 0.6, 0.9, and 0.99. After 4000 iterations, the trend of the optimization target value is as follows: Figure 5 As shown in the figure, the adjustment of the discount factor significantly affects the change and convergence of the optimization target. When the larger value of 0.99 is taken, the final optimization target value can better approach the optimal value due to the focus on past experience; when it is 0.6 and 0.9, the too small discount rate leads to the failure to learn the better action choices in the past, and ultimately fails to complete convergence within the set number of iterations.

[0083] Figure 6 To analyze the stability of the algorithm when the adjustment factor is 2, it can be seen from the loss graph that the algorithm converges after a small number of iterations, which proves the stability of the proposed algorithm. Figure 7This is the performance analysis of the adjustment factor. Due to practical factors, the adjustment multiple should be less than or equal to 4 times. This paper considers three cases of 2 times, 3 times, and 4 times while other parameters remain unchanged. The three cases of adjustment factor values ​​δ = {1 / 4, 1 / 8, 1 / 12}, δ = {1 / 6, 1 / 12, 1 / 18} and δ = {1 / 8, 1 / 16, 1 / 24} are simulated respectively. After 4000 iterations, the trend of the optimization target value change is as follows Figure 7 As shown in the figure, the adjustment of different adjustment factors significantly affects the change and convergence of the optimization target. When the adjustment factor is 4 times, the reward obtained by the agent through learning is greater, which means that the agent pays more attention to the optimal action and converges faster.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An intelligent decision method for physical layer parameters, characterized in that: The following steps are involved: S1: Input the parameters of the RLC layer, MAC layer, and PHY layer as cross-layer information, and use the intelligent scheduler in the MAC to dynamically decide and output the modulation order of the physical layer according to different channel conditions while satisfying the constraints of the upper layer; S2: The MID model models the optimization problem as a Markov decision process, uses the DRL algorithm to design the agent, and defines the state space, action space, and reward function; S3: Using the designed agent, the deep Q network DQN is used to verify the intelligent decision-making scheme for flexible configuration of physical layer parameters in complex network environments; In step S1, the buffer status of RLC, the HOL delay of MAC and the channel quality of PHY are used as cross-layer input information; the cross-layer information is transmitted through the logical channel and the transmission channel, and is used as a dynamically changing network environment to interact with the intelligent agent of MAC in real time; The upper layer constraints in step S1 include whether there is data in the buffer state of the RLC layer and whether the HOL delay time of the MAC layer exceeds the delay threshold; the intelligent scheduler of the MAC layer performs resource scheduling only when there is data in the buffer, and it is invalid when there is no data; the same HOL delay threshold is set for each user request, and the data packets exceeding the HOL delay threshold are discarded from the buffer; the different channel conditions are represented by the signal-to-noise ratio, and the channel quality is divided into three conditions: low, medium and high; The optimization goal of step S1 is to configure the appropriate physical layer modulation order for the current channel environment by defining the maximum number of transmitted symbols under the constraints of buffer status information and HOL delay. The delay constraint only works when there is data. The function is defined as To regulate the delay constraint, the function variable is d t -d th , when the remaining time for the jth requested data packet to be scheduled in the Buffer is less than the HOL delay threshold, it is considered to meet the delay constraint, otherwise it is not satisfied. The mathematical model of the system maximizing the number of symbols is expressed as: Where R m,j (t) represents the number of symbols transmitted by the system, f(d t -d th ) represents the delay constraint function, d t -d th represents the difference between the delay time of the data packet at time t and the delay threshold; st represents the constraint condition, d t represents the delay time of the data packet in the buffer at time t, and the set T = {0, 1, ..., T-1} represents the set of all decision times of the current UE. represents the total channel capacity of the system, Indicates the minimum channel capacity of the system, C1 is the minimum delay limit, that is, to ensure that resources are scheduled under the delay constraint, and C2 is the minimum service requirement of the network, that is, to ensure that the total channel capacity of the current UE cannot be lower than the throughput threshold; In step S2, the MID model designs a Markov decision process MDP model so that the intelligent agent of the MAC layer can decide the physical layer modulation order suitable for the current network environment under different states. The MDP model is designed as follows: First, the state is abstractly represented as the environment, that is, the basis for the agent to make decision modulation orders. The state of MDP at time t is defined as: S(t)={B(t),D(t),H(t)} After observing the environment, the agent obtains the corresponding state S(t) and determines the action A(t) to be performed in this state. The result of the decision is a set of modulation order vectors that satisfy the constraint condition C1. The action space is defined as A(t)=(m(t)) where m(t)={m1(t),m2(t),...,m j (t)|m j (t)∈[4,16,64]},m j (t) represents the modulation order decided by the agent when the UE makes the jth request; Reward is the immediate feedback after the agent takes action in the MDP. The delay constraint function is used as a delay multiplication factor to reward the situation where the constraint is met. If the delay constraint is met, a positive reward is obtained. If it is not satisfied, a negative penalty is obtained. By defining an adjustment factor δ to amplify the reward when the optimal action is taken, when δ is less than 1, the reward is amplified by β times. By amplifying the reward by β times, the agent pays more attention to the optimal action. β is defined as follows: The reward function R(t) is defined according to the defined delay constraints and the state information of the cross-layer parameters. When there is no data to be scheduled in the Buffer, the reward is 0; when there is data to be scheduled in the Buffer, the goal of the system is to accumulate and maximize the positive reward; for low signal-to-noise ratio, the optimal action is low-order modulation, and the adjustment factor is set to be less than 1 when the agent selects the low-order modulation; similarly, the modulation action under medium and high signal-to-noise ratio conditions is considered as the appropriate action for medium and high-order modulation, respectively, and the reward function R(t) is: Where B represents the bandwidth of the system; In step S3, the DQN uses a nonlinear neural network to approximate the state-action value function, extracts features from high-dimensional data, and completes the mapping from state to state-action value. The experience replay pool is responsible for storing the 4-tuple samples obtained by interaction, and works together with the target network and the adaptive estimation optimizer to train the Q network. The Q value function update formula of DQN is as follows: Q * (s,a) is the optimal Q-value function, obtained by recursion. (s(t),a(t),r(t),s(t+1),a(t+1)) means that at the current time t, the system is in state s(t), the immediate reward obtained after executing action a(t), and enters state s(t+1) at the next time with a certain probability to execute action a(t+1). The Q-value function update rule is as follows: Among them, α represents the learning rate, γ∈[0,1] is the discount factor, and the discount factor reflects the network's expectation of obtaining immediate rewards. The experience replay pool is used to store the obtained 4-tuple samples, and small batches of sample data are randomly extracted for training; During the training process, the DQN network trains and updates parameters by minimizing the loss function and gradually approaching the optimal Q function. The loss function is defined based on the difference between the current Q value and the target Q value. The correct direction of the next training step is determined by evaluating this difference. The loss function calculation formula is as follows: Where r(t) is the reward at the current moment; w- and w are the parameters of the target network and the evaluation network respectively; Backward training update of Q network by gradient descent method: Train the DQN model. After training, the optimal weight parameter w of the DQN network is obtained, and the optimal decision modulation strategy is achieved by maximizing the number of symbols in the system.

Images