5G AeroMACS multi-service wireless resource dynamic optimization method

Abstract

The invention relates to the technical field of wireless communication, in particular to a 5G AeroMACS multi-service wireless resource dynamic optimization method, which comprises the following steps of: establishing a 5G AeroMACS-based multi-user NOMA wireless communication system which comprises a 5G AeroMACS base station and a plurality of communication users for real-time communication; determining a multi-objective utility function of wireless resource allocation based on the service priority and the transmission rate of the communication; determining a Markov decision process of interaction between a communication user and a channel environment based on a multi-objective utility function by adopting an action shielding strategy; obtaining the optimal channel selection and communication power of each current communication user; and each communication user adjusts the current channel selection and the communication power in real time, so that the low-complexity real-time resource allocation decision can be realized, and the overall performance of the wireless communication system is improved.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to a 5G AeroMACS multi-service wireless resource dynamic optimization method. Background Art

[0002] 5G AeroMACS is a wireless broadband data link technology designed specifically for airports. It is based on the WiMAX technology standard (IEEE 802.16e) and uses orthogonal frequency division multiple access technology to support efficient data exchange between airports and aircraft, air traffic control and airlines. However, with the rapid development of the global aviation industry, the communication needs of airport ground operations have become increasingly complex and diverse. The dedicated frequency band (5091-5150 MHz) used by AeroMACS faces the problem of limited spectrum resources and cannot meet the growing demand for real-time large-bandwidth communications.

[0003] In order to overcome these limitations, Non-Orthogonal Multiple Access (NOMA) technology is introduced into the airport wireless communication system. NOMA allows multiple users to communicate on the same time-frequency resources and distinguishes the signals of different users at the receiving end through Successive Interference Cancellation (SIC) technology, thereby significantly improving the spectrum efficiency and the number of access users. The application of NOMA technology can not only effectively alleviate the limitations of spectrum resources, but also improve the overall performance of the system by optimizing sub-channel access and power control strategies. However, NOMA technology brings new challenges, namely how to achieve effective resource allocation in a multi-user and high-interference environment. Summary of the invention

[0004] In view of the above problems, the present invention provides a 5G AeroMACS multi-service wireless resource dynamic optimization method, which solves the problem of how to achieve effective resource allocation in a multi-user and high-interference environment in the prior art.

[0005] The present invention provides a 5G AeroMACS multi-service wireless resource dynamic optimization method, comprising the following steps: Step S1, establishing a multi-user NOMA wireless communication system based on 5G AeroMACS, including a 5G AeroMACS base station and multiple communication users; the 5G AeroMACS base station communicates with the communication users in real time; Step S2: determining a multi-objective utility function for wireless resource allocation based on the service priority and transmission rate of the communication; Step S3, adopting an action shielding strategy, based on the multi-objective utility function, determining a Markov decision process of the interaction between the communication user and the channel environment; Step S4, optimizing the Markov decision process by adopting a multi-agent hybrid deep reinforcement learning method based on a dual experience playback mechanism to obtain the optimal channel selection and communication power of each current communication user; Step S5: Each communication user uses the optimized channel selection and communication power to adjust the current channel selection and communication power in real time to complete the dynamic optimization of 5G AeroMACS wireless resources.

[0006] Preferably, in step S1: the channels between the 5G AeroMACS base station and the communication users are evenly divided into sub-channels, there are communication users, the communication user set and the subchannel set are respectively denoted as , The communication users include aircraft, vehicles and sensors on the airport ground, and the priority services are arranged in order from high to low. The service set of a communication user is recorded as , Indicates The total number of services of communication users.

[0007] Preferably, step S2 specifically includes: Step S2-1, based on the requirement of airport communication users to send data packets in order of service priority, determine the average delay of each priority service; Step S2-2, determining the transmission rate of each sub-channel based on the bandwidth and transmission signal-to-noise ratio of the sub-channel selected by the communication user; Step S2-3: determine a multi-objective utility function based on the average delay of each priority service and the transmission rate of each sub-channel.

[0008] Preferably, the average delay expression of each priority service is:

[0009] in, For the Communication user Average latency of priority services, is the business priority index currently being calculated, For the The arrival rate of packets for priority services, It is The business time of each communication user, It is Communication user Utilization of priority service data flows, For the The transmission rate of each communication user; The transmission rate expression of each sub-channel is:

[0010] in, , for The communication user is in Subchannel The transmission rate at the time, is the subchannel bandwidth, For the The communication user is in Subchannel The transmission signal-to-noise ratio at the time; The multi-objective utility function expression is:

[0011] in, For the The communication user is in Subchannel The multi-objective utility function at time.

[0012] Preferably, step S3 specifically includes: Step S3-1, determining a neural network activation function of an action shielding strategy, wherein the neural network activation function makes it so that when a sub-channel is occupied by more than two communication users, other communication users no longer select the sub-channel; Step S3-2: Determine the environment state space, action space and reward of the multi-agent Markov decision process including the action masking strategy.

[0013] Preferably, in step S3-1: The neural network activation function The expression is:

[0014] in, Indicates The number of communication users occupied by the sub-channel; D is the mask activation value; The environmental state space expression of the Markov decision process is:

[0015]

[0016] in, is the state space, For the Communication users The state of the moment, Indicates The communication user is in Subchannel The channel status at the time, and Respectively represent The communication user is in Subchannel The channel status and continuous transmission power at the moment, Indicates Communication users Discrete sub-channel access at time, express Time next A service set of each communication user arranged in descending order of priority; The action space expression of the Markov decision process is:

[0017] in, is the action space, Indicates Discrete sub-channel access for each communication user, Indicates The communication user is in Subchannel Continuous transmission power at the moment; The reward expression of the Markov decision process is:

[0018] in, is a positive real coefficient.

[0019] Preferably, step S4 specifically includes: Step S4-1, constructing an evaluation network and a dual experience replay pool; constructing an action network of each communication user; Step S4-2, based on the reward of the Markov decision process, determining the loss function of the action network based on a clipping method, and determining the loss function of the evaluation network; Step S4-3, the dual experience replay pool receives the channel selection and communication power sent by the communication user; the evaluation network randomly extracts a batch of channel selection and communication power samples from the dual experience replay pool to minimize the loss function of the action network and the evaluation network to update the parameters of the action network and the evaluation network, and sends the updated parameters of the action network to each communication user at a regular time; Step S4-4: Each communication user receives the parameters of the action network, updates the parameters of its own action network, and obtains the most optimized channel selection and communication power.

[0020] Preferably, in step S4-2: The loss function expression of the action network is for:

[0021]

[0022]

[0023]

[0024] in, represents the parameters in the action network, represent The global state of multiple communicating users at time represent The joint actions of multiple communicating users at a given moment, Indicates getting the minimum value. is the probability ratio of the new and old action strategies, Representation strategy In the global state Take joint action Advantages, For strategy Global status The value function of Representation strategy The expected action performed by the agent is Indicates that in a given global state Under the conditions, Indicated in strategy Global status Take joint action The action value of is the steps performed by the agent, Indicates that in a given global state , joint action And strategy Under the conditions, The steps performed by the agent are Expected operation when Restricts the value to Within the range, is a hyperparameter, represents the discount factor; The loss function of the evaluation network is The expression is:

[0025]

[0026] in, represents the parameters in the evaluation network, Data index for the dual experience replay pool The corresponding timing difference error, and They are the data index and data set of the dual experience replay pool respectively; is the penalty factor, for of Regularization term.

[0027] Preferably, in step S4-3, updating the parameters of the action network and the evaluation network by minimizing the loss functions of the action network and the evaluation network specifically includes: The parameter update expressions in the 5G AeroMACS action network and evaluation network are obtained by the gradient descent method, specifically:

[0028]

[0029] in, represents the parameters in the updated action network, Indicates that the value is updated. represents the learning rate of the action network, for The gradient of is the objective function of the optimization strategy, represents the parameters in the updated evaluation network, is the evaluation network learning rate, for The gradient of The parameters in the evaluation network are The joint state and global action are action value.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention introduces the NOMA technology to construct an optimization problem model. By allowing multiple users to share the same time and frequency resources and using the SIC technology at the receiving end to distinguish the signals of different users, the system's spectrum efficiency and the number of access users are significantly improved.

[0031] (2) The present invention provides a multi-objective utility function that comprehensively considers the transmission rate and service priority, and establishes a joint optimization problem model for sub-channel access and power control in the airport uplink communication scenario, which can effectively meet the priority requirements of different service entities and ensure the transmission quality of high-priority services.

[0032] (3) The present invention provides an MDP with an action masking strategy, which effectively reduces the ineffective action exploration in the process of interaction between the intelligent agent and the environment, improves the decision-making accuracy and convergence speed of the algorithm, and helps to achieve efficient real-time decision-making in a complex and dynamically changing communication environment, ensuring the optimal allocation of communication resources and maximizing system performance.

[0033] (4) The present invention adopts an improved deep reinforcement learning algorithm and introduces a dual experience replay mechanism in the MAHDRL algorithm to learn the discrete sub-channel selection and continuous power control strategies respectively, which effectively improves the learning efficiency and the overall performance of the system and optimizes the dynamic allocation strategy of multiple resources in the high-demand airport communication environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0035] Figure 1 This is a flow chart of the 5G AeroMACS multi-service wireless resource dynamic optimization method provided by the present invention.

[0036] Figure 2 This is a schematic diagram of airport mobile communication provided by the present invention.

[0037] Figure 3 This is a schematic diagram of the action shielding provided by the present invention.

[0038] Figure 4 This is a training flow chart of the deep reinforcement learning algorithm provided by the present invention.

[0039] Reference numerals: 11 - 5G AeroMACS base station, 12 - aircraft, 13 - airport vehicle, 14 - sensor, 15 - control tower, 16 - terminal, 17 - runway, 21 - user 1, 22 - user 2, 23 - user 3, 24 - subchannel 1, 25 - subchannel 2, 26 - subchannel K. DETAILED DESCRIPTION

[0040] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0041] The present invention considers factors such as inter-user interference, channel status and transmission power, constructs a joint optimization model for sub-channel access and power control in the airport uplink communication scenario, and establishes a multi-objective utility function based on service priority and transmission rate to ensure that high-priority services get resources first. The MDP containing the action shielding strategy is expressed to improve the algorithm decision accuracy and convergence speed. The MAHDRL algorithm based on the dual experience playback mechanism is used to solve the optimization problem. Each intelligent agent adjusts the sub-channel access and power control strategy in real time according to the state space to achieve efficient dynamic resource allocation.

[0042] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 1 As shown, a 5G AeroMACS multi-service wireless resource dynamic optimization method is disclosed, and the specific implementation steps are as follows: Step S1, establishing a multi-user NOMA wireless communication system based on 5G AeroMACS, including a 5G AeroMACS base station and multiple communication users; the 5G AeroMACS base station communicates with the communication users in real time; the communication users include aircraft, vehicles and sensors on the airport ground.

[0043] In this step, the present invention first determines the 5G AeroMACS multi-user NOMA communication model, and then establishes a joint optimization problem model for its sub-channel access and power control.

[0044] (1) Establish a wireless communication system for uplink multi-user NOMA scenarios based on the 5G AeroMACS framework. Figure 2 As shown in the figure, aircraft and vehicles operate without conflict at the airport. All aircraft, vehicles and sensors on the airport ground are considered communication users and need to upload information to the 5G AeroMACS base station respectively. After completing sub-channel access and power control, each communication user uploads data to the 5G AeroMACS base station to complete individual communication tasks. Sensors are randomly distributed in the airport, and communication users and 5G AeroMACS base stations are equipped with single antennas. The channel is evenly divided into mutually orthogonal sub-channels to avoid inter-channel interference. Assume that there are There are airport communication users, the user set and sub-channel set are respectively denoted as , In addition, the present invention assumes that the positions of the aircraft, vehicle and sensor are fixed in each time slot.

[0045] According to the potential advantages of 5G technology, the NOMA-assisted communication system can enable each subchannel to access multiple airport communication users in the same time slot. Assuming that the communication parameters are The period remains unchanged, i.e. , duration . No. The communication user is in The transmission signal on the sub-channel is denoted as ,satisfy , Indicates calculating the square of the absolute value. The power of a communication user is , then in Time slot The superimposed signal on the sub-channel It can be expressed as: (1) in, Index for communication users, is the total number of communication users, Indicates Is the subchannel in the time slot Assigned to communication users.

[0046] The 5G AeroMACS base station receives direct and reflected components of the communication user signal, and these reflected components are affected by the obstruction of the terminal. The channel model of the present invention is the Rice channel model. The communication user is in The channel gain on the subchannel .

[0047] (2) At the receiving end of the 5G AeroMACS base station, the present invention assumes that the Doppler effect caused by the high maneuverability of aircraft and vehicles can be accurately compensated at the receiver, and the 5G AeroMACS base station can obtain the channel state of each communication user at the beginning of the time slot. The communication user is in The received signal on the subchannel It can be expressed as: (2) in, , , , Respectively represent The communication user is in Whether the subchannels are allocated, channel gain, power and transmission signal; is additive white Gaussian noise with a mean of 0 and a variance of The complex Gaussian distribution of .

[0048] According to the decoding order, The communication user is in SNR on the subchannel It can be expressed as: (3) in, It means calculating the square of the absolute value. is the subchannel additive white Gaussian noise power, Indicates any choice, is a collection of communication time slots.

[0049] Step S2: Determine a multi-objective utility function for wireless resource allocation based on service priority and transmission rate.

[0050] (1) In the present invention, it is assumed that airport communication users can distinguish various types of services. The service set of a communication user is , various communication services are arranged in descending order of priority, Indicates The total number of services of communication users. The communication service data flow of each communication user is modeled as a priority-based queuing model. The arrival process of the data packets received by the communication user conforms to the Poisson distribution, and the average arrival rate is , among which The arrival rate of priority service packets is , then: (4) in, The service priority index.

[0051] In this model, airport communication users send data packets in the order of service priority. For each communication user, the delay of each communication service is related to its priority, data packet size and the transmission rate of the current communication user. The average delay of priority services is: (5) in, For the Communication user Average latency of priority services, is the business priority index currently being calculated, It is The business time of each communication user, It is Communication user Utilization of priority service data flows, For the The transmission rate of each communication user.

[0052] (2) The subchannel bandwidth is set to , then The communication user is in The transmission rate on the subchannel The signal-to-noise ratio can be obtained from formula (6): The calculation result is: (6) (3) In order to ensure that high-priority services can obtain priority resource allocation, considering the service quality requirements of airport communication users in terms of transmission rate and delay, the present invention provides a multi-objective utility function as follows: (7) In order to adapt to the service quality requirements and corresponding service priority requirements of communication users at different airports and achieve optimal resource allocation, the optimization problem is set as: (8) in, Represents any , Optimize for maximum value.

[0053] At the same time, the constraints of the optimization problem are listed. The first is the transmission rate constraint. The transmission rate of the communication user cannot be lower than the transmission rate threshold of the service: (9) in, For the Communication user Minimum transmission rate for priority service data flows.

[0054] The second is the delay constraint. The transmission delay of each type of service cannot exceed the delay threshold of the communication user: (10) in, For the Communication user Average latency of priority services.

[0055] Then there is the power constraint. The total transmission power of the communication users cannot exceed the maximum power limit: (11) in, is the maximum power.

[0056] The last one is the subchannel access constraint, which is divided into whether the subchannel is allocated to the communication user: (12) And the communicating user uses at most one subchannel in each time slot: (13) In addition, each subchannel can be Communication users use: (14) Step S3: Based on the action shielding strategy, determine the Markov decision process of the communication user interacting with the environment.

[0057] (1) Each airport communication user is regarded as an independent decision-making agent, and there is no centralized control unit. Each agent obtains the environmental observation state , the corresponding action selection It can be obtained through the action network, and the joint actions of all agents act on the airport ground communication system. Each agent obtains rewards from the environment after executing the action strategy. And reach the next environmental observation state , while updating the neural network based on reward information from the environment.

[0058] Each agent uses a random strategy to select a subchannel and obtains valid samples by exploring the state space. However, there is a possibility that continuous selection of invalid actions will lead to non-convergence, such as Figure 3 As shown. This problem is solved by giving a certain penalty (i.e., negative reward) in reinforcement learning. Therefore, the present invention provides a neural network activation function that can shield the violation action. Set is the matrix for subchannel selection, where Represents the complex field Matrix, if the subchannel selection Indicates that it has been occupied by more than two communication users, then the subchannel selection neural network output should be replaced by a small coefficient close to 0. Neural network activation function The action-based shielding strategy is designed as follows: (15) in, Indicates The number of communication users occupied by the sub-channel, D is the mask activation value, and the value of D is a small coefficient close to 0, indicating that the sub-channel is no longer suitable for allocation to more communication users.

[0059] (2) For the airport ground NOMA communication system, determine the Markov decision process MDP including the action masking strategy, and define the state space, actions, and rewards of the multi-agent.

[0060] First, define the state space At each time point , will The agent corresponding to the communication user selects The environmental observation state observed when the subchannel include: Indicates The communication user is in Subchannel The channel status at the time, Respectively represent The communication users are on the 1st to the Kth subchannels. The channel gain at time and Respectively represent The communication user is in Subchannel The channel status and continuous transmission power at the moment, Indicates Communication users Discrete sub-channel access at each moment, and a set of services arranged from high to low priority Denoted as: (16) The first The agent corresponding to the communication user selects Subchannel status The set of states is determined as the state space , the expression is: (17) Moreover, each agent can only observe the state space Realize the acquisition of information.

[0061] Next, define the action space Each agent uses the same action space, which contains two parts of the hybrid action space: the discrete subchannel access that the communication user can choose and continuous transmission power Therefore, the mixed action space is recorded as: (18) Finally, define the reward . In multi-agent deep reinforcement learning, the optimal reward obtained by each agent is not completely equivalent to the global optimal reward. Due to the interaction between multiple agents, the action of one agent will affect the rewards obtained by other agents, thereby affecting the global reward. Therefore, the proposed resource allocation problem is considered to be a fully cooperative problem. The global reward is used to replace the reward of each agent itself. The optimization goal is to maximize the long-term return of the airport surface NOMA communication system, so the reward function is defined as: (19) in, is a positive real coefficient. As the training algorithm is iteratively updated in subsequent steps, the agent will tend to choose a strategy that maximizes the long-term cumulative discounted reward. The long-term cumulative discounted reward function is expressed as a weighted accumulation of the agent's action rewards, and the reward weights at subsequent moments gradually decrease over time.

[0062] represent The global state of multiple communicating users at all times, , Respectively represent the first to the Communication users The state of the moment. represents the joint actions of multiple communicating users, Respectively represent the first to the Communication users Action selection at a given moment. In the present invention, communication users can also be represented by intelligent agents.

[0063] Agents adopting continuous power control strategies take joint actions From the global state Transition to the next global state The transition probability Denoted as: (twenty one) in, Indicates time When the global state is given and perform joint actions After that, transfer to the global state The probability density function of Transfer status The microelement of .

[0064] Optimal joint action Through Strategy Get, Strategy The probability distribution is represented by The agent evaluates and optimizes the strategy based on the state value function and the state-action value function: (twenty two) (twenty three) in, For strategy Global status The value function of Representation strategy The expected action performed by the agent is Indicates that in a given global state Under the conditions, Indicated in strategy Global status Take joint action The action value of is the steps performed by the agent, Indicates that in a given global state , joint action And strategy Under the conditions, The steps performed by the agent are Expected operation when .

[0065] The true value is estimated by the Bellman equation value: (twenty four) in, Represents the desired action for the agent to perform.

[0066] In continuous space, by optimizing the strategy To maximize the expected cumulative reward, the state distribution function and strategies Together they determine how the agent chooses actions in each state to maximize its long-term reward. Therefore, the objective function of the optimized policy is Expressed as: (25) in, It is in strategy Global status The distribution function of Indicates the global state The probability distribution of the joint actions of the agents.

[0067] Through the above steps, the present invention determines the MDP including the action masking strategy, and determines the corresponding multi-agent state space, actions and rewards, which can be used in subsequent training iteration steps.

[0068] Step S4: Solve using the MAHDRL algorithm based on the dual experience playback mechanism Based on the MDP of the multi-objective utility function and action shielding strategy based on service priority and transmission rate determined in the previous step, the present invention adopts a multi-agent hybrid deep reinforcement learning (MAHDRL) algorithm based on a dual experience replay mechanism to optimize and solve the wireless resource allocation method of multiple communication users, and obtain the optimal sub-channel access selection and power of each communication user. Each communication user adjusts the sub-channel access and power control strategy in real time according to the state space, and works together to achieve dynamic optimization of sub-channel access and power control.

[0069] (1) The neural network of the resource optimization method provided by the present invention follows the action-evaluation architecture. The base station completes the parameter update of the action network according to the evaluation network, and simultaneously updates the parameters of the evaluation network, aiming to deal with the coordination problem in the multi-agent system. The action network inputs the environmental observation state of each communication user. And output actions according to the current strategy, evaluate the network input global information and output the current state The value of is used to evaluate the value of the current action. After performing action masking, the output value is normalized by the Softmax function to ensure that the sum of the selection probabilities of all channels is 1. After completing the training process, the random strategy is adjusted to a deterministic strategy, corresponding to the action with the highest probability selected based on probability. In a multi-agent environment, each agent has its own action network to decide on actions, while the evaluation network is centralized, evaluating the entire environment state and the behavior of all agents. The action network can be used for distributed execution of each communication user, and the evaluation network can be used for centralized training.

[0070] In addition, the importance sampling technique is introduced to improve sample utilization by dividing the samples collected in a single training process into multiple mini-batches and performing multiple epochs training in each mini-batch to update the action network. At the same time, importance sampling uses another sample with a known distribution to replace the current sample, solving the problem that sample distribution is difficult to obtain.

[0071] On this basis, a dual experience replay mechanism is introduced. By training neural networks for different purposes for different types of actions, it accurately adapts to various control requirements to improve training effects and system performance. The dual experience replay mechanism is designed as two independent experience replay pools, each focusing on different types of action learning: discrete sub-channel selection and continuous power control. The dual experience replay pools store data on continuous transmission power and discrete sub-channel access respectively. The discrete action network uses relevant data, such as service priority and channel occupancy, to optimize channel access and communication user access strategies. The continuous action network uses power control-related data, such as channel gain and interference, to adjust power output. These training data come from their respective dedicated experience replay pools, which can reduce the risk of interference and overfitting between data during the learning process. Figure 4 As shown, the training flowchart of the MAHDRL algorithm using a dual experience replay mechanism is shown.

[0072] (2) The MAHDRL algorithm uses a centralized training and distributed execution scheme. Every time period, 5G AeroMACS base stations broadcast the latest trained neural network parameters, and then update the neural network parameters of each agent simultaneously.

[0073] The action network of the agent is based on the input state By Strategy Output action. Assume strategy For the parameters in the action network is differentiable, then the parameter The gradient of can be expressed as: (26) in, For parameters The gradient of is the optimization objective function in formula (25), It is in strategy Global status The distribution function of . Through the gradient descent method, the parameters in the action network Update the expression to: (27) in, represents the parameters in the updated action network, Indicates that the value is updated. represents the learning rate of the action network, is the timing difference error.

[0074] The evaluation network is responsible for evaluating the environment and outputting The global state-joint action pair And evaluate the parameters in the network Q value at The evaluation network updates its parameters by randomly sampling a batch of samples from the dual experience replay pool to minimize the loss. Data index of the dual experience replay pool The corresponding timing difference error Defined as: (28) in, The data index representing the dual experience replay pool.

[0075] The loss function of the evaluation network is expressed as: (29) in, A data set for the dual experience replay pool.

[0076] Evaluate the parameters in the network through the gradient descent method Update the expression to: (30) in, represents the parameters in the updated evaluation network, is the evaluation network learning rate, For parameters gradient.

[0077] (3) The system gradually adjusts the target network parameters through a soft update mechanism, balancing the exploration of new strategies with the use of known strategies to ensure the stability and high performance of the learning process. In addition, the system also regularly evaluates the effectiveness of the strategy to ensure the maximum throughput and energy efficiency of the communication system.

[0078] In order to prevent the gap between the new and old strategies from being too large due to the large update of the action, the present invention provides an action network loss function in the form of a Clip function: (31) in, Indicates getting the minimum value. is the probability ratio of the new and old action strategies, ,in, express Under the new strategy Under global status conditions The probability of joint actions, express Under the old strategy Under global status conditions The probability of joint actions, Representation strategy The agent is in the global state Take joint action Advantages, , Restricts the value to Within the range, It is a hyperparameter usually set to 0.2. By clipping to eliminate values ​​outside the interval, the minimum value of the target is finally obtained.

[0079] In addition, the loss function of the evaluation network is added Regularization term is used to prevent the parameters from overfitting the training data and to increase the model's ability to handle abnormal data. The loss function is modified to: (32) in is the penalty factor used to adjust The influence of the regularization term, for of Regularization term.

[0080] The present invention proposes a 5G AeroMACS multi-service wireless resource dynamic optimization method. The method adopts the NOMA architecture and combines deep reinforcement learning technology to solve the spectrum resource constraint problem in the airport wireless communication system by constructing a joint optimization problem model of sub-channel access and power control, and meet the communication needs of multiple services. Specifically, by constructing a joint optimization model of sub-channel access and power control, factors such as interference between communication users, channel status and transmission power are comprehensively considered to achieve efficient utilization of spectrum resources; a multi-objective utility function based on service priority and transmission rate is established to ensure that high-priority services can obtain resource allocation first, thereby improving the overall performance of the system; an MDP containing an action shielding strategy is expressed, each communication user is regarded as an intelligent agent, the communication environment state, agent action selection and reward function are defined, the decision-making process and interaction relationship of communication users are described, and invalid action shielding is designed to reduce invalid sample exploration, thereby improving the decision accuracy and convergence speed of the algorithm; MAHDRL based on the dual experience replay mechanism is used for solution, each agent adjusts the sub-channel access and power control strategy in real time according to the state space, and the agents work together to achieve dynamic optimization of sub-channel access and power control, ultimately achieving efficient and dynamic resource allocation decisions for the airport wireless communication system.

[0081] Although the specific embodiments of the present invention have been described in a specific order, each action or step should be understood to require that such action or step be performed in the specific order shown or in a sequential order, or require that all illustrated actions or steps should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0082] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A 5G AeroMACS multi-service wireless resource dynamic optimization method, characterized in that: The following steps are involved: Step S1, establishing a multi-user NOMA wireless communication system based on 5G AeroMACS, including a 5G AeroMACS base station and multiple communication users; the 5G AeroMACS base station communicates with the communication users in real time; Step S2: determining a multi-objective utility function for wireless resource allocation based on the service priority and transmission rate of the communication; Step S3, adopting an action shielding strategy, based on the multi-objective utility function, determining a Markov decision process of the interaction between the communication user and the channel environment; Step S4, optimizing the Markov decision process by adopting a multi-agent hybrid deep reinforcement learning method based on a dual experience playback mechanism to obtain the optimal channel selection and communication power of each current communication user; Step S5: Each communication user uses the optimized channel selection and communication power to adjust the current channel selection and communication power in real time to complete the dynamic optimization of 5G AeroMACS wireless resources.

2. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 1, characterized in that: In step S1: The channels between the 5G AeroMACS base station and the communication users are evenly divided into sub-channels, there are communication users, the communication user set and the subchannel set are respectively denoted as , The communication users include aircraft, vehicles and sensors on the airport ground, and the priority services are arranged in order from high to low. The service set of a communication user is recorded as , Indicates The total number of services of communication users.

3. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 2, characterized in that: Step S2 specifically includes: Step S2-1, based on the communication user's requirement to send data packets in order of service priority, determine the average delay of each priority service; Step S2-2, determining the transmission rate of each sub-channel based on the bandwidth and transmission signal-to-noise ratio of the sub-channel selected by the communication user; Step S2-3: determine a multi-objective utility function based on the average delay of each priority service and the transmission rate of each sub-channel.

4. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 3 is characterized in that: The average delay expression of each priority service is: in, For the Communication user Average latency of priority services, is the business priority index currently being calculated, For the Communication user The arrival rate of packets for priority services, It is The business time of each communication user, It is Communication user Utilization of priority service data flows, For the The transmission rate of each communication user; The transmission rate expression of each sub-channel is: in, For the The communication user is in Subchannel The transmission rate at the time, is the subchannel bandwidth, For the The communication user is in Subchannel The transmission signal-to-noise ratio at the time; The multi-objective utility function expression is: in, For the The communication user is in Subchannel The multi-objective utility function at time.

5. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 4, characterized in that: Step S3 specifically includes: Step S3-1, determining a neural network activation function of an action shielding strategy, wherein the neural network activation function makes it so that when a sub-channel is occupied by more than two communication users, other communication users no longer select the sub-channel; Step S3-2: Determine the state space, action space and reward of the multi-agent Markov decision process including the action masking strategy.

6. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 5, characterized in that: In step S3-1: The neural network activation function The expression is: in, Indicates The number of communication users occupied by the sub-channel; D is the mask activation value; The state space expression of the Markov decision process is: in, is the state space, For the Communication users The state of the moment, Indicates The communication user is in Subchannel The channel status at the time, and Respectively represent The communication user is in Subchannel The channel status and continuous transmission power at the moment, Indicates Communication users Discrete sub-channel access at time, express Time next A service set for each communication user; The action space expression of the Markov decision process is: in, is the action space, Indicates Communication users Discrete sub-channel access at time, Indicates The communication user is in Subchannel Continuous transmission power at the moment; The reward expression of the Markov decision process is: in, is a positive real coefficient.

7. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 6, characterized in that: Step S4 specifically includes: Step S4-1, constructing an evaluation network and a dual experience replay pool; constructing an action network of each communication user; Step S4-2, based on the reward of the Markov decision process, determining the loss function of the action network based on a clipping method, and determining the loss function of the evaluation network; Step S4-3, the dual experience replay pool receives the channel selection and communication power sent by the communication user; the evaluation network randomly extracts a batch of channel selection and communication power samples from the dual experience replay pool to minimize the loss function of the action network and the evaluation network to update the parameters of the action network and the evaluation network, and sends the updated parameters of the action network to each communication user at a regular time; Step S4-4: Each communication user receives the parameters of the action network, updates the parameters of its own action network, and obtains the most optimized channel selection and communication power.

8. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 7, characterized in that: In step S4-2: The loss function expression of the action network is for: in, represents the parameters in the action network, Indicates getting the minimum value. is the probability ratio of the new and old action strategies, represent The global state of multiple communicating users at time represent The joint actions of multiple communicating users at a given moment, Indicated in strategy Global status Take joint action Advantages, For strategy Global status The value function of Representation strategy The expected action performed by the agent is Indicates that in a given global state Under the conditions, Indicated in strategy Global status Take joint action The action value of is the steps performed by the agent, Indicates that in a given global state , joint action And strategy Under the conditions, The steps performed by the agent are Expected operation when Restricts the value to Within the range, is a hyperparameter, represents the discount factor; The loss function of the evaluation network is The expression is: in, represents the parameters in the evaluation network, Data index for the dual experience replay pool The corresponding timing difference error is and They are the data index and data set of the dual experience replay pool respectively; is the penalty factor, for of Regularization term.

9. The 5G AeroMACS multi-service wireless resource dynamic optimization method according to claim 8, characterized in that: In step S4-3, updating the parameters of the action network and the evaluation network by minimizing the loss functions of the action network and the evaluation network specifically includes: The parameter update expressions in the 5G AeroMACS action network and evaluation network are obtained by the gradient descent method, specifically: in, represents the parameters in the updated action network, Indicates that the value is updated. represents the learning rate of the action network, for The gradient of is the objective function of the optimization strategy, represents the parameters in the updated evaluation network, is the evaluation network learning rate, for The gradient of The parameters in the evaluation network are The global state and joint action are action value.

Images