Marine wireless network resource allocation method based on attention mechanism

Abstract

The invention relates to a maritime wireless network resource allocation method based on an attention mechanism, and belongs to the technical field of wireless communication. Comprising the following steps: (1) an action suppression method for controlling action output of an intelligent agent: restraining switching action output by adopting a mode of setting an output sensitive value and a sensitive value attenuation coefficient for the action output; 2) a resource allocation-oriented attention perception method: on the basis of inputting a large amount of environmental information, dividing observation information of nodes into self-state information and environmental state information, and performing targeted processing on information with different characteristics to obtain a mixed attention weight; and 3) a deep reinforcement learning method based on a potential energy reward function: adopting a DDQN-based deep reinforcement learning method and the potential energy-based reward function to obtain access selection action output. According to the invention, the switching ping-pong effect can be inhibited, the node service matching performance is enhanced, and the node service transmission efficiency is improved.

Description

Technical Field

[0001] The invention relates to a method for allocating wireless network resources at sea based on an attention mechanism, and belongs to the technical field of wireless communications. Background Art

[0002] With the continuous development of communication technology, human activities are gradually expanding from land to sea, and there is an urgent need for wide-area full-time communication guarantee at sea to provide uninterrupted network services for space-based, air-based, ground-based and marine information services. However, heterogeneous wireless networks in marine environments have a wide coverage range and complex structure, and have diverse network resources, service characteristics and quality of service (QoS) requirements. At the same time, limited communication resources and environmental uncertainty have greatly increased the difficulty of human production and activities. There is an urgent need to arrange and allocate resources with different network dynamic characteristics to maximize the use of limited communication resources.

[0003] At present, the research on resource allocation of wireless networks mainly focuses on three aspects: power allocation, spectrum allocation and access selection. Radio access technology (RAT), as the key technology that enables users to connect seamlessly under its wide area coverage, is the first step in resource scheduling and allocation. At present, access selection technology is mainly divided into two categories. One is to rely on traditional operations research methods as technical support to accurately calculate the access results. The other is to rely on machine learning methods to find the optimal access strategy. Traditional operations research methods are often used to solve low-dimensional and discrete problems. The air, land and sea scenes are complex, with many constraints and a large solution space. Traditional operations research methods are difficult to fully cope with. When facing high-dimensional and complex problems, it is necessary to rely on machine learning methods to provide function approximation methods. Although the existing technology has solved the problem of land wireless network resource allocation from different angles, it is difficult to directly migrate from the existing land and air wireless network resource allocation scheme to the sea scene due to the fast node movement speed, wide wireless network area, and many types and differences in business requirements in the sea scene. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the above-mentioned prior art and provide a method for allocating wireless network resources at sea based on an attention mechanism. The method includes an action suppression method for controlling the action output of an intelligent agent, an attention perception method for resource allocation, and a deep reinforcement learning method based on a potential energy reward function. The method can suppress the switching ping-pong effect, enhance the matching of node services, and improve the transmission efficiency of node services.

[0005] The present invention is a method for allocating wireless network resources at sea based on an attention mechanism, which is special in that it includes:

[0006] 1. Action suppression method for controlling the action output of the intelligent agent: the output sensitivity value and sensitivity value attenuation coefficient are set for the action output to constrain the switching action output;

[0007] 2. Attention perception method for resource allocation: Based on the input of a large amount of environmental information, the observation information of the node is divided into self-state information and environmental state information, and information with different characteristics is processed in a targeted manner to obtain mixed attention weights;

[0008] 3. Deep reinforcement learning method based on potential reward function: A deep reinforcement learning method based on DDQN and a potential-based reward function are used to obtain access selection action output.

[0009] Preferably, the step 1 comprises the following steps:

[0010] 1) According to the action value output, set the optimal action value difference threshold at adjacent moments. The value of the Q network output action at time t is Q t , the value of the output action at the previous moment is Q t-1 , set the sensitivity value Q s for:

[0011] |Q t -Q t-1 |≥Q s (9)

[0012] 2) As the algorithm output gradually stabilizes, the sensitivity value Q s Decay over time to adapt the output of the algorithm, setting the sensitivity value Q s The decay rate is ε s,t , ε s,t The decay coefficient is μ, and the action value constraint function (10) of the selective inhibition unit that allows the agent to switch links is obtained. If the condition is not met, the current connection action is maintained:

[0013]

[0014] Preferably, the step 2 specifically comprises the following steps:

[0015] 1) First, the "receptive field" of the agent is split into self-state and environment state. The self-state includes the connection network parameters and other state parameters of the agent at the current moment, and the environment state includes the network parameters and distance of the connectable base station;

[0016] 2) For the self-state information, a linear perceptron is used to map each state parameter according to its dimension to obtain a sub-weight, calculate the correlation between the node's own attributes and the environmental parameters, and normalize these correlations into attention weights. The agent sub-weight at time t is expressed as:

[0017]

[0018] Then all sub-weights are weighted to obtain the self-attention weight for selecting the business type under self-state perception, that is

[0019]

[0020] 3) For the environmental state information, a multi-layer perceptron is used to calculate the parameter similarity, and the implicit information of the state is mined through multiple fully connected layers. Finally, the spatiotemporal attention weights of the business type under the environmental state sense are output, which is expressed as:

[0021]

[0022] 4) After obtaining the self-attention weight and spatiotemporal attention weight, the self-attention is used as the benchmark, the spatiotemporal attention is used for error correction, and the weights are multiplied proportionally to finally obtain the recalibrated weight of the business type feature, that is, the mixed attention weight, which is expressed as:

[0023]

[0024] Preferably, step 3 comprises the following steps:

[0025] 1) Calculate the business type selection weight of the current resource pool from the environment output state value through the attention perception method, and execute the business selection action The environment state value is obtained through the potential energy-based reward function, and the reward function expression is:

[0026]

[0027] in It is the main line reward, which is rewarded according to the business completed by the agent at the current moment; It is an auxiliary reward function that rewards or punishes the agent based on the network parameters of the network that it previously connected to.

[0028] 2) Combine the agent's service list information and the environment state value and input them into the Q network to calculate the action value of the agent's optional actions at the current moment, which is expressed as:

[0029]

[0030] 3) Obtaining the agent network access selection strategy by controlling the action suppression method of the agent action output Execute the selection strategy and store the obtained environment state value in the experience pool, repeating steps 1)-3) until the agent stops.

[0031] Compared with the prior art, the method for allocating wireless network resources at sea based on the attention mechanism of the present invention has the following beneficial effects:

[0032] 1. The motion suppression method for controlling the motion output of an intelligent agent provided by the present invention can solve the problem of a ping-pong effect when the intelligent agent outputs motions.

[0033] 2. The attention perception method for resource allocation provided by the present invention can enhance the state perception ability of the intelligent body and solve the problem of mismatch between characteristics of the network, nodes and services.

[0034] 3. The wireless network resource allocation method provided by the present invention can jointly allocate the service access and network access selection of the node, thereby improving the service completion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a principle block diagram of the marine wireless network resource allocation method based on the attention mechanism;

[0036] Figure 2 This is a principle block diagram of the attention perception method for resource allocation;

[0037] Figure 3 It is a comparison chart of the total number of switching times of different technical methods;

[0038] Figure 4 It is a comparison chart of the business completion status of different technical methods;

[0039] Figure 5 This is a comparison chart of the round average network parameters of different technical methods;

[0040] Figure 6 It is a comparison chart of the completion of business priorities of different technical methods;

[0041] Figure 7 It is a comparison chart of the business type completion status of different technical methods. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0043] A method for allocating wireless network resources at sea based on an attention mechanism in this embodiment includes the following steps:

[0044] 1. Action suppression method for controlling the action output of the intelligent agent: the output sensitivity value and sensitivity value attenuation coefficient are set for the action output to constrain the switching action output;

[0045] Step 1 includes the following steps:

[0046] 1) According to the action value output, set the optimal action value difference threshold at adjacent moments. The value of the Q network output action at time t is Q t , the value of the output action at the previous moment is Q t-1 , set the sensitivity value Q s for:

[0047] |Q t -Q t-1 |≥Q s (17)

[0048] 2) As the algorithm output gradually stabilizes, the sensitivity value Q s Decay over time to adapt the output of the algorithm, setting the sensitivity value Q s The decay rate is ε s,t , ε s,t The decay coefficient is μ, and the action value constraint function (10) of the selective inhibition unit that allows the agent to switch links is obtained. If the condition is not met, the current connection action is maintained:

[0049]

[0050] 2. Attention perception method for resource allocation: Based on the input of a large amount of environmental information, the node's observation information is divided into self-state information and environmental state information, and information with different characteristics is processed in a targeted manner to obtain mixed attention weights;

[0051] Step 2 specifically includes the following steps:

[0052] 1) First, the "receptive field" of the agent is split into self-state and environment state. The self-state includes the connection network parameters and other state parameters of the agent at the current moment, and the environment state includes the network parameters and distance of the connectable base station;

[0053] 2) For the self-state information, a linear perceptron is used to map each state parameter according to its dimension to obtain a sub-weight, calculate the correlation between the node's own attributes and the environmental parameters, and normalize these correlations into attention weights. The sub-weight of the agent at time t is expressed as:

[0054]

[0055] Then all sub-weights are weighted to obtain the self-attention weight for selecting the business type under self-state perception, that is

[0056]

[0057] 3) For the environmental state information, a multi-layer perceptron is used to calculate the parameter similarity, and the implicit information of the state is mined through multiple fully connected layers. Finally, the spatiotemporal attention weights of the business type under the environmental state sense are output, which is expressed as:

[0058]

[0059] 4) After obtaining the self-attention weight and spatiotemporal attention weight, the self-attention is used as the benchmark, the spatiotemporal attention is used for error correction, and the weights are multiplied proportionally to finally obtain the recalibrated weight of the business type feature, that is, the mixed attention weight, which is expressed as:

[0060]

[0061] 3. Deep reinforcement learning method based on potential reward function: A deep reinforcement learning method based on DDQN and a potential-based reward function are used to obtain access selection action output.

[0062] Step 3 includes the following steps:

[0063] 1) Calculate the business type selection weight of the current resource pool from the environment output state value through the attention perception method, and execute the business selection action The environment state value is obtained through the potential energy-based reward function, and the reward function expression is:

[0064]

[0065] in It is the main line reward, which is rewarded according to the business completed by the agent at the current moment; It is an auxiliary reward function that rewards or punishes the agent based on the network parameters of the network that it previously connected to.

[0066] 2) Combine the agent's service list information and the environment state value and input them into the Q network to calculate the action value of the agent's optional actions at the current moment, which is expressed as:

[0067]

[0068] 3) Obtaining the agent network access selection strategy by controlling the action suppression method of the agent action output Execute the selection strategy and store the obtained environment state value in the experience pool, repeating steps 1)-3) until the agent stops.

[0069] in accordance with Figure 1 , Figure 2,According to the processing steps of the marine wireless network resource allocation method based on the attention mechanism, the relevant parameters are set as follows: the number of agents is 1, the speed of high-speed nodes is [15,350] m / s, the speed of base stations is [1,15] m / s, the learning rate of the Q network is 1e-4, the number of network layers is 4, the number of neurons is 300 / 200 / 60 / 24, the sampling time is 1s, the discount factor is 0.95, the batch sample size is 600, the experience buffer length is 1.2e5, the smoothing factor is 1e-5, and the target network update frequency is 5. Under the above simulation conditions, the present invention is simulated and verified according to the specific modulation and demodulation steps.

[0070] The flowchart of the whole method is as follows Figure 1 As shown in Figure 1, the overall method is divided into three modules, of which the first module is the DDQN algorithm part, which will not be described here. The second module is the attention perception module. The environment output state value is first transmitted to the attention perception module to obtain the business type selection weight of the current resource pool and execute the business selection action. The agent service list information and the environment state value are combined and input into the Q network, and the action is output. The third module is the selection inhibition module. The action output by the Q network and its action value are first detected by the action inhibition module. If the conditions are met, they are directly output. If the conditions are not met, the action output at the previous moment replaces the current network output, and finally outputs the network access selection action. In order to compare and analyze the performance difference between the present invention and the prior art, a simulation analysis was carried out using a marine wireless network as a scenario. The present invention is named A-DDQN, and the comparison techniques are deep reinforcement learning algorithm DDPG (LILLICRAP TP, HUNT JJ, PRITZELA, elt. Continuous control with deep reinforcement learning [C / OL] / / International Conference on Learning Representations (ICLR). 2016.), SAC (HAARNOJA T, ZHOU A, HARTIKAINEN K, elt. Soft Actor-Critic Algorithms and Applications [C / OL] / / Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018: 1861-1870.), DDQN (VAN HASSELT H, GUEZ A, SILVERD. Deep Reinforcement Learning with Double Q-learning [C / OL] / / Proceedings of the AAAI Conference on Artificial Intelligence. AAAI, 2016, 30: 2094-2100.), the traditional method D-AHP (Mao Zhongyang, Zhang Zhilin, Liu Xiguo, et al. Maritime mobile node network selection algorithm based on dynamic AHP [J]. Systems Engineering and Electronics, 2022, 44(06): 2011-2018) and the greedy algorithm are compared. The specific details of the scenario are as follows: In addition to the intelligent agent itself, there are other greedy nodes with the same task as the intelligent agent in the simulation scenario. The order in which all nodes receive services from the resource pool is sorted in real time according to the uniform distribution. The algorithm performance test scenario is a random scenario. The randomness is reflected in the continuous random high-speed movement of nodes, the continuous random low-speed movement of base stations, the random generation of resource pool service attribute values, and the random interference of network parameters. The traditional method and the greedy algorithm do not require training, so they are directly used in the test scenario for testing. Due to the strong randomness of the scenario, the performance of the following indicators is the average value of every 20 rounds. The multiple links of each base station in the scenario are regarded as heterogeneous wireless communication networks. The user's service quality is mainly affected by four network parameter indicators: packet delay, packet jitter, bandwidth, and packet loss rate. It is assumed that the base station has the ability to meet the user's communication network switching and access requests, and the switching time and cost are negligible.Under the combined effects of the four basic network parameters and random environmental factors, the user's service transmission rate will vary to varying degrees. This closed-loop interaction constitutes the interaction between the network and the user. The base stations follow the Poisson distribution. Services with different demands are placed in the resource pool and wait for nodes to access them. Services are divided into three categories: conversational, interactive, and streaming. Their contents are (1) real-time location, maritime management, planned route information, etc. (2) maritime distress information, emergency support information, etc. (3) hydrological information, sensor raw signals, maps, etc. The average bandwidth of the three services is 3, 12, and 50 MHz, respectively, and the duration is 8, 16, and 24 seconds, respectively, accounting for 33%, 34%, and 33% of the total. Services are also divided into three different priorities to distinguish the importance, with level 1 being the largest and level 3 being the smallest.

[0071] 1. Total number of switching times

[0072] The statistical standard for the number of handovers is defined as follows: if the access selection strategies at time t and time t+1 are different, the number of handovers is increased by one, and the total number of handovers in one complete round is accumulated. Figure 3 As can be seen in the figure, the selection inhibition unit has a good effect. Compared with the original algorithm DDQN, the proposed algorithm has reduced the number of switches by 69.23%. Although it is slightly higher than other deep learning algorithms under the switch number constraint, the number of switches only accounts for 10% of the total action time, which is reasonable and acceptable. Compared with the original standard algorithm, the algorithm with the selection inhibition unit has the number of switches controlled within 10% of the total action time under the action of the selection inhibition unit. This fully proves the versatility of the selection inhibition unit and can effectively control the frequent switching of the algorithm. In order to fit the actual situation, the comparison algorithms of the proposed algorithms will target the algorithms with the number of switches constrained within 10% of the total action time, namely the DDPG_ls, SAC_ls and A-DDQN_ls algorithms.

[0073] 2. Business completion quantity and value

[0074] Figure 4 (a) shows the total amount of business completed in each round of each algorithm. It can be seen that the proposed algorithm has increased the total amount of completion by about 6.4% and 11.73% compared with the DDQN_ls and DDPG_ls algorithms. At the same time, the ablation experiment shows that the attention perception module of the A-DDQN algorithm increases the number of business completed by 22.7%. This shows that the attention perception module of the proposed method can still effectively improve the number of business completed when the action switching is constrained, because the attention perception module better matches the node status and business needs, so that the node business queue remains unobstructed. Combined with Figure 3 and Figure 4(a) shows that after constraining the number of switching times, the number of completed services of the SAC algorithm is reduced by 11.37%, and that of the DDQN algorithm is reduced by 13.53%. Although the selection of the suppression unit can effectively suppress the number of switching times, some optimal actions are sacrificed at the cost of reduced service transmission efficiency.

[0075] Figure 4 (b) shows the total amount of unfinished business in each round of each algorithm. It can be seen that the total amount of timeout business of the proposed algorithm is reduced by 24.56% and 4.4% respectively compared with the DDQN_ls algorithm and the SAC_ls algorithm. This is because the business transmission efficiency of the three algorithms is lower than the normal transmission standard in some time periods. However, the proposed method matches the node status and business. When the transmission efficiency is low, it matches the short-duration business, which can reduce some losses.

[0076] Figure 4 (c) represents the total value of completed services. Since the agent has a limited time to execute services, the total value of services completed in a round is capped. The total value of services completed by the proposed algorithm is increased by 13.72% and 6.57% respectively compared with the other two algorithms. It can be seen that the perception attention module plays a certain role in adapting network parameters and selecting service types. The module makes full use of the network to which it is connected and maximizes transmission efficiency. That is, when selecting non-highest priority services, it adopts the strategy of selecting short-duration, low-value services under low network parameters and long-duration, high-value services under high network parameters.

[0077] 3. Average network parameters

[0078] Figure 5 A comparison chart of the average network parameters of the agent rounds is given. The average network parameter refers to the mean value of each network parameter in each round, which can reflect the quality of the algorithm's access strategy within the round. In the current scenario, packet delay, packet loss and transmission rate are in the range of (0,70], (10,45] and (1,2] respectively, which are good transmission efficiency intervals. It can be seen from the figure that the three network parameters of the proposed algorithm are in the range of good transmission efficiency in 80% of the rounds, which is about 8%-15% more than other comparison methods. This means that the agent under the proposed method can complete more business in a resource-constrained environment and better adapt to a strong dynamic environment.

[0079] 4. Business priority completion quantity

[0080] Figure 6 The completion status of different business priorities is compared. The proposed algorithm is improved by 6.1% and 11.73% compared with the DDQN_ls and SAC_ls algorithms, respectively, which shows that the proposed method has a high priority for the completion of high-value business. Figure 4It can be seen that compared with other algorithms, the proposed method achieves a virtuous cycle of completing more business, updating the list more frequently, and completing a greater total business value while abandoning low-value business.

[0081] (5) Business type completion quantity

[0082] Figure 7 (a) is a comparison chart of the number of completed type I businesses. The proposed algorithm improves 8.7% and 14.83% compared to the DDQN_ls and SAC_ls algorithms, respectively. Figure 4 From the above, the strategy of the intelligent agent to select services is to select as many conversational and interactive services as possible, and only when the network parameters are good enough will a small number of large-bandwidth flow services be selected for execution, which is in line with the basic strategy of maximizing the value of service completion. The SAC algorithm learns various solutions to the problem through maximum entropy. Compared with other algorithms that can only find one optimal path, it is more robust to dynamic environments, but does not distinguish the adaptability of different services and nodes. The algorithm based on the traditional method has low transmission efficiency and slow service list update. When it receives services from the resource pool, most of the resources in the resource pool are the services left by other nodes in the past, so more interactive and flow services are selected. The proposed algorithm plays the role of perceiving and adapting the compatibility of nodes and services through the attention perception module, that is, it perceives the state of itself and the environment in real time, and adjusts the action strategy to find the optimal solution in each state. In summary, although the number of optimal performance of the proposed algorithm is not the largest, the overall performance is the best. While suppressing the access selection ping-pong effect, it improves the total number and total value of service completion. The performance parameters of the algorithm are compared as shown in the following table.

[0083] Table 1 Algorithm performance comparison

[0084]

[0085] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for allocating wireless network resources at sea based on attention mechanism, characterized in that The following steps are involved: 1) Action suppression method for controlling the action output of the intelligent agent: the output of the switching action is constrained by setting the output sensitivity value and the sensitivity value attenuation coefficient for the action output; 2) Attention-aware method for resource allocation: Based on the input of a large amount of environmental information, the node's observation information is divided into self-state information and environmental state information, and information with different characteristics is processed in a targeted manner to obtain mixed attention weights; 3) Deep reinforcement learning method based on potential reward function: A deep reinforcement learning method based on DDQN and a potential-based reward function are used to obtain access selection action output.

2. A method for allocating wireless network resources at sea based on an attention mechanism according to claim 1, characterized in that The step 1 comprises the following steps: 1) According to the action value output, set the optimal action value difference threshold at adjacent moments. The value of the Q network output action at time t is Q t , the value of the output action at the previous moment is Q t-1 , set the sensitivity value Q s for: |Q t -Q t-1 |≥Q s (1) 2) As the algorithm output gradually stabilizes, the sensitivity value Q s Decay over time to adapt the output of the algorithm, setting the sensitivity value Q s The decay rate is ε s,t , ε s,t The decay coefficient is μ, and the action value constraint function (10) of the selective inhibition unit that allows the agent to switch links is obtained. If the condition is not met, the current connection action is maintained:

3. A method for allocating wireless network resources at sea based on an attention mechanism according to claim 1, characterized in that The step 2 specifically includes the following steps: 1) First, the "receptive field" of the agent is split into self-state and environment state. The self-state includes the connection network parameters and other state parameters of the agent at the current moment, and the environment state includes the network parameters and distance of the connectable base station; 2) For the self-state information, a linear perceptron is used to map each state parameter according to its dimension to obtain a sub-weight, calculate the correlation between the node's own attributes and the environmental parameters, and normalize these correlations into attention weights. The sub-weight of the agent at time t is expressed as: Then all sub-weights are weighted to obtain the self-attention weight for selecting the business type under self-state perception, that is 3) For the environmental state information, a multi-layer perceptron is used to calculate the parameter similarity, and the implicit information of the state is mined through multiple fully connected layers. Finally, the spatiotemporal attention weights of the business type under the environmental state sense are output, which is expressed as: 4) After obtaining the self-attention weight and spatiotemporal attention weight, the self-attention is used as the benchmark, the spatiotemporal attention is used for error correction, and the weights are multiplied proportionally to finally obtain the recalibrated weight of the business type feature, that is, the mixed attention weight, which is expressed as:

4. A method for allocating wireless network resources at sea based on an attention mechanism according to claim 1, characterized in that The step 3 comprises the following steps: 1) Calculate the business type selection weight of the current resource pool from the environment output state value through the attention perception method, and execute the business selection action The environment state value is obtained through the potential energy-based reward function, and the reward function expression is: in It is the main line reward, which is rewarded according to the business completed by the agent at the current moment; It is an auxiliary reward function that rewards or punishes the agent based on the network parameters of the network that it previously connected to. 2) Combine the agent's service list information and the environment state value and input them into the Q network to calculate the action value of the agent's optional actions at the current moment, which is expressed as: 3) Obtaining the agent network access selection strategy by controlling the action suppression method of the agent action output Execute the selection strategy and store the obtained environment state value in the experience pool, repeating steps 1)-3) until the agent stops.

Images